Aero-engine dense component three-dimensional instance segmentation method and system

Abstract

A kind of aero-engine dense component three-dimensional instance segmentation method and system, including in aero-engine complex body point cloud, the point cloud of dense component cluster is obtained by semantic segmentation and clustering;The three-dimensional candidate area of component instance is generated by detecting the near neighbor point and center offset of component instance center;Center feature of each point in three-dimensional candidate area is constructed, and it is aggregated with low-level geometry and high-level semantic feature of each point;High-level point center feature is generated using hierarchical feature learning and decoding neural network module, and three-dimensional component instance mask prediction and scoring double branch are constructed to obtain instance mask in three-dimensional component candidate area.The present application can realize the fine three-dimensional instance segmentation of dense component under the whole machine state of aero-engine, resist the interference of complex body structure to component segmentation, and the center feature constructed has stronger feature representation when distinguishing dense component points.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional vision, in particular to an aero-engine dense component three-dimensional instance segmentation method and system. BACKGROUND

[0002] The aero-engine, as the power source of aircraft and other aircraft, has high complexity and precision, and is an important embodiment of the industrial level of a country. Due to the long-term high temperature, high pressure and strong vibration during the operation of the aero-engine, in order to ensure the operation stability, each process from the design, processing to assembly of the aero-engine has very strict requirements. After the overall assembly of the aero-engine is completed, a series of strict body and component detection needs to be carried out. However, the assembled components on the whole machine have the characteristics of large number, small size and dense distribution, and often the assembly distance between multiple same components is about 1 cm or even only a few millimeters, which brings great challenges to the efficient and accurate individual detection of components. At present, the individual positioning and detection of dense components under the assembly of the aero-engine mainly rely on the design model of the engine and the manual visual method. However, manual detection has the problems of low efficiency and limited detection range. Due to the differences in professional experience and subjectivity of the detection personnel, there are certain degree of missed detection and false detection problems in the detection results. Therefore, the automatic instance segmentation of dense components in complex large-size bodies is an important prerequisite and key link for subsequent assembly quality detection of components, and has important significance for improving the detection efficiency and accuracy of dense components.

[0003] In the existing industrial intelligent detection scene, the machine vision method based on two-dimensional images is often used to carry out automatic target recognition. However, the two-dimensional image method has the problem of data essence in the instance segmentation task of dense components, because the two-dimensional image loses the real geometric structure and depth information in the single-view imaging process, and these information is very important for segmenting dense targets and then realizing one-by-one detection. Three-dimensional data can provide rich geometric structure, shape and scale information. Various high-precision three-dimensional scanners such as handheld and flow type are more and more used to collect three-dimensional point clouds of industrial products or scenes, and it has great application value to carry out multi-dimensional attribute recognition and high-precision measurement of components under the three-dimensional point cloud data mode.

[0004] A kind of aviation engine clamp fine-grained three-dimensional identification method (patent publication number: CN113936218A), it is proposed that a kind of aviation engine clamp fine-grained type identification method based on three-dimensional point cloud modal, the method is positioned by locating the clamp parts on the three-dimensional point cloud of aviation engine, then constructs fine-grained point cloud feature learning neural network model and automatically learns the fine-grained shape and structural attribute of each clamp, to realize the type classification of clamp.This method has good effect on the recognition of the clamp point cloud with minor size difference or local shape and structure difference, however, there is still certain extension space in the instance recognition problem of densely distributed parts cluster. In 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) meeting, Vu et al. proposed "SoftGroup for 3D Instance Segmentation on Point Clouds", this kind of three-dimensional point cloud instance segmentation method first regresses the offset of each point relative to the center of the instance, then uses the semantic category of each point to enhance the clustering effect of instance point cloud, which avoids the influence of false semantic segmentation result on instance segmentation, however, for densely distributed objects, the center offset of the edge point often exists to some extent. Deviation, prone to cause over-segmentation problem, the existing three-dimensional point cloud instance segmentation method and its application effect on the dense parts segmentation task of aviation engine still need to be improved. SUMMARY

[0005] For the instance segmentation problem of densely distributed parts on the complex body of aviation engine, the present application provides a three-dimensional instance segmentation method and system for dense parts of aviation engine, specifically, a three-dimensional instance segmentation method and system for dense parts of aviation engine based on center feature, the present application automatically learns the point features related to the center of dense parts instance by constructing hierarchical point cloud feature learning neural network module and multi-task branch module, realizes the automatic segmentation of dense parts, adapts to the interference of complex body structure of aviation engine on small size part feature learning and recognition, and can realize automatic instance segmentation of full range parts in whole machine assembly state.

[0006] The application firstly uses a three-dimensional scanning device to perform three-dimensional scanning on an aero-engine whole machine to be inspected, and obtains three-dimensional point clouds of the whole machine; a three-dimensional point cloud semantic segmentation deep neural network model is used for point cloud semantic segmentation to obtain three-dimensional point clouds of parts; density-based point cloud clustering is performed on the three-dimensional point clouds of the parts to obtain three-dimensional point clouds of dense part clusters; for the point clouds of each part cluster, the binary neighborhood and three-dimensional offset of each point relative to the center of the instance to which the point belongs, and the more detailed two-class semantic category of each point are detected, the center of each part instance point cloud is calculated, and a three-dimensional candidate region of the part instance is generated; in each candidate region, the mask of the part instance is detected and the evaluation score of the mask is regressed, and finally the part instance segmentation in the complex body of the aero-engine is realized.

[0007] The technical scheme of the application is as follows:

[0008] The application provides an aero-engine dense part three-dimensional instance segmentation method, which comprises the following steps:

[0009] Step S100, three-dimensional scanning is performed on an aero-engine whole machine to be inspected, and three-dimensional point clouds are obtained; a three-dimensional point cloud semantic segmentation deep neural network model is used for semantic segmentation with the part to be inspected as the foreground and the complex body as the background, three-dimensional point clouds of the parts are obtained, and density-based point cloud clustering is performed on the three-dimensional point clouds of the parts to obtain three-dimensional point cloud clusters of the dense parts;

[0010] Step S200, a backbone neural network module is constructed on each dense part three-dimensional point cloud for fine-grained hierarchical point high-level semantic feature learning; a three-branch multi-task learning neural network module is constructed to detect the binary neighborhood and three-dimensional offset of each point relative to the center of the instance to which the point belongs, and the two-class semantic category of each point;

[0011] Step S300, the center of each part instance point cloud in the dense part cluster is calculated according to the offset, fine-grained semantic category and center neighborhood label of each point obtained in step S200, a three-dimensional candidate region in which each part instance is located is generated according to the average size of the same type of part, and three-dimensional instance detection of the part in the complex body is realized;

[0012] Step S400, the center features of each three-dimensional point in the three-dimensional candidate region of each part are constructed, and then the center features of each point are aggregated with the low-level geometric features of each point and the high-level semantic features learned by the backbone neural network;

[0013] Step S500, a hierarchical feature learning and feature decoding neural network module is constructed to perform hierarchical encoding and decoding on the aggregated features, obtain the mask of the part instance in each three-dimensional candidate region, and regress the evaluation score of the mask, and realize instance segmentation of the dense part in the three-dimensional point cloud of the complex aero-engine body.

[0014] Further, the step S100 comprises:

[0015] Step S110, semantic segmentation is performed on the three-dimensional point cloud of the aero-engine whole machine, so as to take the segmented parts as foreground and the complex body as background, obtain the three-dimensional point cloud of various types of parts, and remove the influence of the remaining body three-dimensional point cloud of the aero-engine on the positioning and segmentation of the parts instances;

[0016] Step S120, point cloud clustering based on point density is performed on the point cloud of various types of parts to obtain the point cloud cluster of the same type of parts.

[0017] Further, the step S200 comprises:

[0018] Step S210, a point cloud feature learning backbone neural network module is constructed to extract the high-level semantic features of each point;

[0019] Step S220, a multi-task branch is constructed, including a near neighbor point detection branch of the instance center, an offset regression branch, and a fine semantic segmentation branch.

[0020] Further, in the step S220, based on the point-level high-level semantic features learned by the backbone network, the following tasks are respectively performed on each dense part point cloud cluster: (1) detecting the binary near neighbor property of each point relative to the center of the belonging instance; (2) regressing the three-dimensional offset of each point relative to the center of the belonging instance; (3) detecting the more fine two-class semantic category of each point.

[0021] Further, the step S300 comprises:

[0022] Step S310, for each dense part point cloud cluster, first, the instance center near neighbor points and the center offset vector thereof are determined according to the center near neighbor property labels of the instance points in the cluster, and the center offset vector of each point is used to offset and collapse the near neighbor points to the center of the belonging instance in the Euclidean space;

[0023] Step S320, based on the breadth-first search method, the collapsed center near neighbor points are clustered, the spatial coordinates of the three-dimensional near neighbor points clustered into one class are averaged to obtain the three-dimensional coordinates of the instance center to which the near neighbor points belong;

[0024] Step S330, according to the fine semantic category of the points detected by the third branch, a three-dimensional candidate region to which each part instance belongs is defined on the original point cloud which has not been offset and collapsed, each region takes the instance center as the region center, and the three-dimensional part instance detection in the complex body represented by the three-dimensional candidate region is realized.

[0025] Further, the step S400 comprises: constructing the center feature of each three-dimensional point in the three-dimensional candidate region of each part, and performing feature aggregation on the low-level point geometric feature and the high-level point semantic feature learned by the backbone neural network module to form the fusion feature of each point, which represents the multi-dimensional attribute of each point.

[0026] Further, the step S500 comprises: constructing a point cloud level feature learning and feature decoding module to perform deep feature encoding and mapping decoding on the point fusion feature of the three-dimensional candidate region, and then constructing a two-task branch, one branch of which is used to detect whether each point in the candidate region is a part instance belonging to the candidate region, so as to obtain a mask of the part instance; and the other branch is used to regress the evaluation score of the instance mask, and finally the two branches jointly realize the instance segmentation of the dense part in the complex aero-engine body three-dimensional point cloud.

[0027] The application further provides an aero-engine dense part three-dimensional instance segmentation system, comprising:

[0028] The three-dimensional point cloud cluster acquisition module is used for three-dimensional scanning of the aero-engine to be detected, acquiring three-dimensional point clouds, taking the part to be detected as the foreground and the complex body as the background, performing semantic segmentation by using a three-dimensional point cloud semantic segmentation deep neural network model, acquiring part three-dimensional point clouds, and performing density-based point cloud clustering on the part three-dimensional point clouds to acquire three-dimensional point cloud clusters of the dense part.

[0029] The construction module is used for constructing a backbone neural network module to learn the high-level semantic feature of the point in a fine-grained level on each three-dimensional point cloud of the dense part, and constructing a three-branch multi-task learning neural network module to detect the binary neighbor and three-dimensional offset of each point relative to the center of the instance to which the point belongs, and the two-class semantic category of each point.

[0030] The three-dimensional instance detection module is used for calculating the center of each part instance point cloud in the dense part cluster according to the offset, fine-grained semantic category and center neighbor label of each point, generating a three-dimensional candidate region in which each part instance is located according to the average size of the same part, and realizing three-dimensional instance detection of the part in the complex body.

[0031] The feature aggregation module is used for constructing the center feature of each three-dimensional point in the three-dimensional candidate region of each part, and then performing feature aggregation on the low-level geometric feature of each point and the high-level semantic feature learned by the backbone neural network.

[0032] An instance segmentation module, a hierarchical feature learning and feature decoding neural network module encodes and decodes the aggregated features, obtains a mask of a part instance in each three-dimensional candidate region, and regresses an evaluation score of the mask, so as to realize instance segmentation of dense parts in a complex aero-engine body three-dimensional point cloud.

[0033] Another aspect of the present application provides an electronic device, comprising:

[0034] at least one processor; and,

[0035] a memory connected in communication with the at least one processor; wherein,

[0036] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method.

[0037] The present application also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method.

[0038] Compared with the prior art, the present application can achieve the following beneficial technical effects:

[0039] (1) The present application is directly based on aero-engine three-dimensional point cloud, which can effectively utilize real depth and geometric structure information under the three-dimensional point cloud data mode, avoiding the problem of target scale and distance loss due to perspective projection imaging in two-dimensional image analysis method, especially for dense target instance detection with small gap between target instances.

[0040] (2) The present application adopts a coarse-to-fine technical route, first learns coarse-grained point cloud features to perform overall engine three-dimensional point cloud semantic segmentation to remove the interference of complex body structure, and then learns fine-grained point cloud features to further refine the boundary point attribution of part instance, to realize fine-grained part point cloud semantic segmentation.

[0041] (3) The present application constructs and learns multi-dimensional low-level geometric and high-level semantic fusion deep point cloud features by perceiving the relative position relationship of three-dimensional part points relative to the center of the belonging instance, enhances the feature representation ability of dense part instance points, and can effectively segment out the densely distributed part instances; the present application is suitable for the application conditions of large and complex aero-engine overall assembly. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The overall implementation flowchart of the present application is shown in the figure;

[0043] Figure 2The point cloud diagram of the densely distributed clamp parts on the local body of the aero-engine of the application;

[0044] Figure 3 The three-dimensional instance detection technical roadmap of the densely distributed parts of the application;

[0045] Figure 4 The three-dimensional instance mask segmentation technical roadmap of the densely distributed parts of the application. DETAILED DESCRIPTION

[0046] To make the purpose, technical solutions and advantages of the application more clear and explicit, the application is further described in detail below in combination with specific embodiments and with reference to the drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the application. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the application.

[0047] The application comprises semantic segmentation and clustering of point clouds of densely distributed parts in aero-engine complex body point clouds; generating three-dimensional candidate regions of part instances by detecting the near neighbor points and center offset of the part instance center; constructing the center features of each point in the three-dimensional candidate region, and aggregating them with the low-level geometric and high-level semantic features of each point; generating high-level point center features using hierarchical feature learning and decoding neural network modules, and constructing three-dimensional part instance mask prediction and scoring double branches to obtain instance masks in three-dimensional part candidate regions.

[0048] The application is further described in detail below in combination with specific embodiments.

[0049] Specifically, as shown in Figure 1

[0050] Step S100, using a three-dimensional scanning device to perform three-dimensional scanning on the whole aero-engine to be inspected, and obtaining a three-dimensional point cloud; using a three-dimensional point cloud semantic segmentation deep neural network model to perform semantic segmentation with the to-be-inspected parts as the foreground and the complex body as the background, and obtaining a three-dimensional point cloud of the parts; then, performing density-based point cloud clustering on the three-dimensional point cloud of the parts, and obtaining a three-dimensional point cloud cluster of densely distributed parts;

[0051] Step S200, constructing a backbone neural network module to perform fine-grained hierarchical point high-level semantic feature learning on each three-dimensional point cloud of the densely distributed parts; constructing a three-branch multi-task learning neural network module to respectively detect the binary neighbor relationship and three-dimensional offset of each point relative to the center of the instance to which the point belongs, and the more fine-grained two-class semantic category of each point;

[0052] ​Step S300, according to the offset, fine-grained semantic category and center proximity label of each point obtained in step b, the center of each part instance point cloud in the dense part cluster is calculated, and a three-dimensional candidate region where each part instance is located is generated according to the average scale of the same part, so as to realize three-dimensional instance detection of the part in the complex machine body;

[0053] Step S400, for each three-dimensional candidate region of the part, the center feature of each three-dimensional point in the region is constructed, and then the center feature of each point is aggregated with the low-level geometric feature of each point and the high-level semantic feature learned by the backbone neural network;

[0054] Step S500, the hierarchical feature learning and feature decoding neural network module is constructed to perform hierarchical encoding and decoding on the aggregated feature, obtain the mask of the part instance in each three-dimensional candidate region, and regress the evaluation score of the mask, so as to realize instance segmentation of the dense part in the three-dimensional point cloud of the complex aero-engine machine body;

[0055] Further, the step S100 comprises:

[0056] Step S110, semantic segmentation is performed on the three-dimensional point cloud of the aero-engine whole machine, taking the part to be segmented as foreground and the complex machine body as background, to obtain three-dimensional point clouds of various types of parts, and remove the influence of the three-dimensional point cloud of the remaining machine body of the aero-engine on the positioning and segmentation of the part instance.

[0057] Step S120, point cloud clustering based on point density is performed on various types of part point clouds to obtain point cloud clusters of the same type of part, and each point cloud cluster contains one or more densely distributed parts of the same type.

[0058] Further, the step S200 comprises:

[0059] A point cloud feature learning backbone neural network module is constructed to extract high-level semantic features of each point, and then a multi-task branch is constructed, including a proximity point detection branch of the instance center, an offset regression branch, and a fine semantic segmentation branch. Based on the point-level high-level semantic features learned by the backbone network, each branch performs the following tasks on each dense part point cloud cluster: (1) detecting the binary proximity of each point relative to the center of the instance to which it belongs; (2) regressing the three-dimensional offset of each point relative to the center of the instance to which it belongs; (3) detecting the more fine-grained two-class semantic category of each point.

[0060] Step S210, center proximity point detection branch: this branch is composed of a multi-layer perceptron and a Softmax layer, and the point high-level semantic features of the dense part point cloud cluster are input to obtain the center proximity score of each point p i ​This represents the probability that the i-th point in a dense cluster of component point clouds is a nearest neighbor of the center of a certain component. In the branch inference process, if The predicted center k nearest neighbor label c of the i-th point i If it is 1, otherwise c i =0. Use the cross-entropy loss function L center To supervise the testing branches near the center, L center The definition is as follows:

[0061]

[0062] Where n represents the number of points in the dense point cloud cluster of components. Let represent the truth value of the nearest neighbor property of the i-th point.

[0063] Step S220, Center Offset Regression Branch: This branch uses a multilayer perceptron to regress the center offset of each point p from the high-level point features of the dense component point cloud. i The offset vector o relative to the center of its instance i Using the loss function L offset To supervise the center offset regression branch, L offset The definition is as follows:

[0064]

[0065] Where n represents the number of points in the dense component point cloud; It is a binary indicator that shows whether the i-th point in the dense component point cloud is an instance point of a component. For p i The truth value of the instance center to which the point belongs, ||·||1 represents the L1 norm of a vector.

[0066] Step S230, Instance Point Fine-grained Semantic Segmentation Branch: This branch uses a multilayer perceptron to regress the high-level point features of the dense component point cloud to detect a more refined binary semantic category for each point, i.e., instance point or background point, to further remove background points from the dense component point cloud. This branch also consists of a multilayer perceptron and a Softmax layer, first regressing each point p i semantic score Let be the probability that the i-th point in the dense component point cloud is an instance point. During training, the cross-entropy loss function L is used. semantic To supervise the fine semantic segmentation branch of instance points, as shown in the following formula.

[0067]

[0068] Where n represents the number of points in the dense component point cloud; the semantic label truth value of the i-th point, denotes that the i-th point is an instance point, denotes that the i-th point is a background point. In the branch inference process, if the fine semantic class label s i of the i-th point = 1, otherwise s i = 0.

[0069] Further, the step S300 comprises:

[0070] Step S310, for each dense part point cloud cluster, first determine the instance center near neighbor points and the center offset vector thereof according to the center near neighbor label of each instance point in the cluster, and offset collapse the near neighbor points in the Euclidean space to the instance center to which the near neighbor points belong by using the center offset vector of each point;

[0071] Step S320, based on the breadth-first search method, cluster the collapsed center near neighbor points, and take the average of the spatial coordinates of the three-dimensional near neighbor points clustered into a class to obtain the three-dimensional coordinates of the instance center to which the near neighbor points belong;

[0072] Step S330, according to the point fine semantic class detected in the third branch in step S200, define a three-dimensional candidate region to which each part instance belongs on the original point cloud which has not been offset collapsed, and each region takes the instance center as the region center, thereby realizing the three-dimensional part instance detection in the complex machine body represented by the three-dimensional candidate region;

[0073] Further, the step S400 comprises, for each three-dimensional candidate region of a part, constructing the center feature of each three-dimensional point in the region, aggregating the center feature with the low-level point geometric feature and the high-level point semantic feature learned by the backbone neural network module to form a fusion feature of each point, and representing the multi-dimensional attributes of each point.

[0074] Specifically, it comprises:

[0075] Step S410, constructing the point-level center feature of each three-dimensional point in the three-dimensional candidate region relative to the region center oc, which represents the relative position relationship between each point and the instance center to which the point belongs, and the feature is used to further distinguish whether each point in the region belongs to the instance corresponding to the current region center. The explicit encoding process of the center feature rp i of each point is as follows,

[0076]

[0077] wherein denotes the dimension concatenation operation, oc denotes the three-dimensional candidate region center, and ||·||2 denotes the L2 norm of a vector.

[0078] Step S420, multi-level point cloud feature fusion: after obtaining the center features of each three-dimensional point in the candidate region, multi-level point cloud feature fusion is performed on the features to be fused, including the point semantic features f i , three-dimensional coordinates p i , three-dimensional normal vector n i , fused point feature f' i The dimension of the fused feature is d1+10, where d1 is the dimension of the point semantic feature.

[0079]

[0080] Step S430, hierarchical learning of fused features: after obtaining the fused features of each point in the instance candidate region, a hierarchical feature encoding and feature decoding network module is constructed to perform hierarchical learning on the point features of the instance candidate region, thereby forming high-level fused features.

[0081] Further, the step S500 includes: constructing a point cloud hierarchical feature learning and feature decoding module to perform deep feature encoding and mapping decoding on the point fused features of the three-dimensional candidate region, and then constructing a two-task branch, one branch of which is used to detect whether each point in the candidate region is a candidate region belonging to the part instance, thereby obtaining a mask of the part instance; and the other branch is used to regress the evaluation score of the instance mask, and finally the two branches jointly realize instance segmentation of the dense part in the complex aero-engine body three-dimensional point cloud.

[0082] Specifically, after obtaining the high-level fused features of each three-dimensional point in the instance candidate region, a double-branch module is constructed to obtain the instance mask and its score in the instance candidate region from top to bottom, and the double-branch module includes a three-dimensional instance mask prediction branch and a mask evaluation branch.

[0083] (1) The three-dimensional instance mask prediction branch divides each point in the instance candidate region into two categories: points belonging to the instance corresponding to the center of the current region and points belonging to other instances. The high-level fused features of each point are mapped to a feature with a dimension of 2 via a multi-layer perceptron, and then the two-dimensional feature of each point is mapped to [0, 1] via a Softmax layer, thereby obtaining the probability of each point belonging to the current instance and other instances (i is the index of the point), if m i =1, otherwise m i =0, thereby obtaining the mask of the current instance. The cross-entropy loss function L mask is used to supervise the three-dimensional instance mask prediction branch, as shown in the following formula.

[0084]

[0085] where q is the number of three-dimensional instance candidate regions in the point cloud of the dense part cluster, and ne is the number of points of the e-th instance candidate region, is the mask ground truth of the i-th point in each instance candidate region.

[0086] (2) The three-dimensional instance mask evaluation branch: this branch scores the instance mask obtained by the three-dimensional instance mask prediction branch, uses the max-pooling layer to fuse the high-level features of the points belonging to the current instance in each three-dimensional candidate region, then uses the multi-layer perception to map to a feature with a dimension of 1x1, and uses the sigmod function to normalize the feature to [0, 1] to obtain the score of the instance mask.

[0087] Specific embodiments of the present application:

[0088] Step S1: use a three-dimensional scanning device to perform three-dimensional scanning on the whole aero-engine to be inspected, and obtain three-dimensional point cloud; in this embodiment, a GOM ATOS ScanBox 8160 large three-dimensional scanning system is used to obtain the three-dimensional point cloud data of the whole machine. The three-dimensional scanning system is composed of two high-precision three-dimensional scanning heads, two mechanical arms and corresponding guide rails, the scanning head realizes three-dimensional reconstruction of a single frame of image based on a binocular camera and a stripe projection, and the three-dimensional point cloud of the whole machine is obtained through global registration; the three-dimensional point cloud semantic segmentation deep neural network model is used for semantic segmentation with the part to be inspected as the foreground and the complex machine body as the background, and the three-dimensional point cloud of the part is obtained; then, the three-dimensional point cloud of the part is subjected to point cloud clustering based on density, and the three-dimensional point cloud of the dense part cluster is obtained; Figure 2 A cluster of aero-engine densely distributed clamp parts is shown, which contains 6 clamps with an assembly gap of about 1-2 cm between them;

[0089] Step S2: construct a point cloud feature learning backbone neural network module to extract the high-level semantic features of each point, and in this embodiment, a sparse convolutional neural network is used as the backbone network, which first voxelizes the three-dimensional point cloud data with a voxel size of 0.25 mm, extracts the features of the three-dimensional voxels, and then restores the high-level semantic features of each original point through a feature decoding network module; then, a multi-task branch is constructed, including a near neighbor point detection branch of the instance center, an offset regression branch, and a fine semantic segmentation branch, and each branch learns the point-level high-level semantic features based on the backbone network to perform the following tasks on each dense part point cloud cluster: (1) detecting the binary neighborhood of each point relative to the center of the instance to which the point belongs; (2) regressing the three-dimensional offset of each point relative to the center of the instance to which the point belongs; (3) detecting the more fine two-class semantic categories of each point, and the overall structure diagram is as shown in Figure 3 .

[0090] (1) Center near neighbor point detection branch: this branch is composed of a multi-layer perception and a Softmax layer, inputs the high-level semantic features of the points of the dense part point cloud cluster, and obtains the binary neighborhood of each point p relative to the center of the instance to which the point belongs, that is, the binary value of the distance between the point and the center of the instance to which the point belongs is obtained, and the value is 1 when the distance is less than a certain threshold, and the value is 0 when the distance is greater than the certain threshold.i the center proximity score of the i-th point denotes the probability that the i-th point of the dense part point cloud cluster is a proximity point of some part center, In the branch inference process, if the predicted center k-proximity label c i of the i-th point is 1, otherwise c i = 0. The center proximity point detection branch is supervised using a cross-entropy loss function L center , which is defined as follows center

[0091]

[0092] where n denotes the number of points in the dense part point cloud cluster, denotes the proximity ground truth of the i-th point. In this embodiment, 500 points around the part instance center are defined as proximity points, i.e., k = 500

[0093] (2) Center offset regression branch: This branch uses a multi-layer perceptron to regress the offset vector o i of each point p i relative to the instance center it belongs to from the high-level point features of the dense part point cloud. The center offset regression branch is supervised using a loss function L offset , which is defined as follows offset

[0094]

[0095] where n denotes the number of points in the dense part point cloud; is a binary indicator that indicates whether the i-th point in the dense part point cloud is an instance point of a part, is the ground truth of the instance center to which the p i point belongs, and ||·||1 denotes the L1 norm of a vector.

[0096] (3) Instance point fine-grained semantic segmentation branch: This branch uses a multi-layer perceptron to detect the more fine-grained binary semantic class of each point, i.e., instance point or background point, from the high-level point features of the dense part point cloud, for further removing background points in the dense part point cloud. This branch is also composed of a multi-layer perceptron and a Softmax layer, which first regresses the semantic score s i of each point p is the probability that the i-th point in the dense part point cloud is an instance point, In the training process, a cross-entropy loss function L semantic is used to supervise the instance point fine-grained semantic segmentation branch, L​​semantic is defined as follows.

[0097]

[0098] wherein n represents the number of points in the dense zero-part component point cloud; is the semantic label true value of the i-th point, represents that the i-th point is an instance point, represents that the i-th point is a background point. In the branch inference process, if the fine semantic class label s i of the i-th point is 1, otherwise s i = 0.

[0099] Step S3: According to the offset of each point, the fine-grained semantic class and the center neighbor point label obtained in step S2, the center of each zero-part component instance point cloud in the dense zero-part component cluster is calculated, including the following sub-steps,

[0100] (1) For each dense zero-part component point cloud cluster, first determine the instance center neighbor point and its center offset vector according to the center neighbor property label of each instance point in the cluster, and collapse the neighbor point to the instance center in the Euclidean space by using the center offset vector of each point;

[0101] (2) Based on the breadth-first search method, the collapsed center neighbor points are clustered, the spatial coordinates of the three-dimensional neighbor points clustered into a class are averaged to obtain the three-dimensional coordinates of the instance center to which the neighbor points belong. In this embodiment, the neighbor point clustering radius is selected as 2mm, and the clustering point number threshold is 5, that is, when the number of points clustered into a class is less than 5, the class is regarded as a noise class and is automatically filtered out;

[0102] (3) According to the fine semantic class of the points detected in the third branch in step b, define a three-dimensional candidate region to which each zero-part component instance belongs on the original point cloud without offset collapse, and take the instance center as the region center, so as to realize the three-dimensional instance detection of the zero-part components in the complex machine body represented by the three-dimensional candidate region. According to the average size of the zero-part components, generate a three-dimensional candidate region in which each zero-part component instance is located, and realize the three-dimensional instance detection of each zero-part component in the complex machine body. In this embodiment, the radius of the candidate region is set to 30mm according to the average size of the clamp component;

[0103] Step S4: For each three-dimensional candidate region of the zero-part component, construct the center feature of each three-dimensional point in the region, and then aggregate the center feature of each point with the low-level geometric feature of each point and the high-level semantic feature learned by the backbone neural network, such as Figure 4 the instance center perception feature representation module in the embodiment.

[0104] (1) Construct the center feature of each three-dimensional point in the three-dimensional candidate region The point-level center feature relative to the center oc of the region characterizes the relative position relationship between each point and the center of the instance to which it belongs, and is used to further distinguish whether each point in the region belongs to the instance corresponding to the current region center. The center feature rp of each point i The explicit encoding process is as follows,

[0105]

[0106] wherein represents a dimension concatenation operation, oc represents a three-dimensional candidate region center, and ||·||2 represents an L2 norm of a vector.

[0107] (2) Multi-level point cloud feature fusion: after obtaining the center feature of each three-dimensional point in the candidate region, multi-level point cloud feature fusion is performed. The fused features include the point semantic feature f i learned by the backbone network module, the three-dimensional coordinates p i , the three-dimensional normal vector n i , and the fused point feature f' i The fused feature has a dimension of d1+10, wherein d1 is the dimension of the point semantic feature.

[0108]

[0109] (3) Hierarchical learning of fused features: after obtaining the fused features of each point in the instance candidate region, a hierarchical feature encoding and feature decoding network module is constructed to perform hierarchical learning on the point features of the instance candidate region, thereby forming high-level fused features.

[0110] Step S5: after obtaining the high-level fused features of each three-dimensional point in the instance candidate region, a double-branch module is constructed to obtain the instance mask and its score in the instance candidate region from top to bottom. The double-branch includes a three-dimensional instance mask prediction branch and a mask evaluation branch. The overall technical solution of this step and step S5 is shown in Figure 4 .

[0111] (1) The three-dimensional instance mask prediction branch divides the points in the instance candidate region into two categories: points belonging to the instance corresponding to the current region center and points belonging to other instances. The high-level fused features of each point are mapped to a feature with a dimension of 2 via a multi-layer perceptron, and then the two-dimensional feature of each point is mapped to [0, 1] via a Softmax layer, thereby obtaining the probability of each point belonging to the current instance and other instances (i is the index of the point). If , then m i = 1, otherwise m i = 0, thereby obtaining the mask of the current instance. The cross-entropy loss function L mask is used to supervise the three-dimensional instance mask prediction branch, as shown below.

[0112]

[0113] wherein q is the number of three-dimensional instance candidate regions in the point cloud of the dense cluster of zero parts, n e is the number of points of the e-th instance candidate region, is the mask ground truth of the i-th point in each instance candidate region.

[0114] (2) The three-dimensional instance mask evaluation branch: this branch scores the instance mask obtained by the three-dimensional instance mask prediction branch, uses a max-pooling layer to fuse the high-level features of the points belonging to the current instance in each three-dimensional candidate region, then uses a multi-layer perception to map to a feature with a dimension of 1x1, and uses a Sigmod function to normalize the feature to [0, 1] to obtain the score of the instance mask.

[0115] In order to evaluate the performance of the method of the application, experiments were carried out on the real three-dimensional point cloud of an aero-engine, and the average precision of the method of the application in performing three-dimensional instance segmentation of dense clamp parts was tested, and the quantitative results are shown in Table 1. Among them, the index average accuracy APx represents the intersection over union threshold of the part instance and the true value. The quantitative evaluation index is defined as follows: for a clamp part instance mask M i segmented by the method of the application, if the intersection over union IoU between M i and the corresponding true mask is greater than the threshold x, and the IoU is the maximum value in the intersection over union of all segmented instance masks {M i}(i = 1,..., n M )(where n M is the total number of segmented instance masks, and i is the index of the instance mask) and the true mask , the instance is recorded as a true positive TP, otherwise as a false positive FP; the accuracy p is calculated according to , and the recall r is calculated according to By changing the score threshold of the segmented clamp instance, the true positive rate-recall rate curve under multiple scores can be obtained, and the area between this curve and the two perpendicular coordinate axes is the average accuracy APx under the current intersection over union threshold x. AP avg The index represents the average value of all AP scores with a step of 0.05 and an IoU threshold from 0.5 to 0.9. From the quantitative results, it can be seen that the method of the application has higher average accuracy than the optimal method SoftGroup (published in CVPR2022) at the same period on the dense clamp three-dimensional point cloud data of the aero-engine.

[0116] Table 1 Average accuracy of the method of the application in three-dimensional instance segmentation of the dense clamp of the aero-engine

[0117] Method / evaluation criteria AP avg ±std]]> AP 90 ±std AP 75 ±std]]> AP 50 ±std]]> SoftGroup 0.877±0.059 0.691±0.14 0.895±0.074 0.939±0.025 Method of the invention 0.905±0.045 0.736±0.067 0.931±0.047 0.949±0.035

[0118] The application further provides an aero-engine dense component three-dimensional instance segmentation system, comprising:

[0119] The three-dimensional point cloud cluster of the dense component is obtained by performing three-dimensional scanning on the aero-engine to be detected, obtaining three-dimensional point clouds, taking the component to be detected as the foreground and the complex body as the background, performing semantic segmentation on the three-dimensional point clouds by using a three-dimensional point cloud semantic segmentation deep neural network model, obtaining three-dimensional point clouds of the component, and performing point cloud clustering based on density on the three-dimensional point clouds of the component.

[0120] The construction module is used for constructing a backbone neural network module on each three-dimensional point cloud of the dense component to learn fine-grained hierarchical point high-level semantic features, and constructing a three-branch multi-task learning neural network module to detect the binary neighbor and three-dimensional offset of each point relative to the center of the instance to which the point belongs and the two-class semantic category of each point.

[0121] The three-dimensional instance detection module is used for calculating the center of the three-dimensional point cloud of each component instance in the dense component cluster according to the offset, fine-grained semantic category and center neighbor label of each point, generating a three-dimensional candidate region in which each component instance is located according to the average scale of the same component, and realizing three-dimensional instance detection of the component in the complex body.

[0122] The feature aggregation module is used for constructing the center feature of each three-dimensional point in the three-dimensional candidate region of each component, and then performing feature aggregation on the center feature of each point and the low-level geometric feature of each point and the high-level semantic feature learned by the backbone neural network.

[0123] The instance segmentation module is used for constructing a hierarchical feature learning and feature decoding neural network module to perform hierarchical encoding and decoding on the aggregated features, obtaining the mask of the component instance in each three-dimensional candidate region, and regressing the evaluation score of the mask, so as to realize instance segmentation of the dense component in the three-dimensional point cloud of the complex aero-engine body.

[0124] The application further provides an electronic device, comprising at least one processor and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method.

[0125] The application further provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to execute the method.

[0126] In summary, the present application provides an aero-engine dense component three-dimensional instance segmentation method and system, comprising: three-dimensional point cloud scene semantic segmentation and instance initial clustering to obtain dense component cluster three-dimensional point cloud; learning high-level point semantic features of each cluster and detecting binary neighbor property and three-dimensional offset of each point relative to the instance center, and more detailed two-class semantic categories of each point; calculating the instance center of the dense component and generating instance three-dimensional candidate region; constructing center features for each point of each candidate region and performing low-level-high-level feature fusion learning; detecting component instance fine three-dimensional mask and regressing mask evaluation score. The method of the present application is tested on the three-dimensional point cloud data of the aero-engine whole machine after final assembly, and can realize three-dimensional instance segmentation of dense clamp components in the case of complex structure machine body as background, with an average accuracy of 0.905±0.045, which is superior to the three-dimensional point cloud instance segmentation method of the same period. The method of the present application has good resistance to complex machine body background, the point center feature and the fusion learning feature thereof have stronger feature expression ability for dense instance points, can realize instance segmentation of dense components, and is suitable for the application conditions of large and complex aero-engine whole machine final assembly. The technical scheme of the present application can realize fine three-dimensional instance segmentation of dense components in the state of aero-engine whole machine, resist the interference of complex machine body structure on component segmentation, and the center feature constructed has stronger feature representation ability in distinguishing dense component points.

[0127] It should be understood that the above specific embodiments of the present application are only used for illustrative or explanatory purposes of the principles of the present application, and do not constitute a limitation of the present application. Therefore, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the present application shall be included in the protection scope of the present application. In addition, the appended claims of the present application are intended to cover all variations and modifications falling within the scope and boundary of the appended claims, or the equivalent forms of such scope and boundary.

Claims

1. An aeroengine dense part three-dimensional instance segmentation method, characterized in that, The method comprises the following steps: Step S100, three-dimensional scanning is performed on the to-be-inspected aero-engine whole machine to obtain three-dimensional point clouds, semantic segmentation is performed on the to-be-inspected parts by using a three-dimensional point cloud semantic segmentation deep neural network model, taking the to-be-inspected parts as foreground and the complex machine body as background, three-dimensional point clouds of the parts are obtained, and the three-dimensional point clouds of the parts are subjected to point cloud clustering based on density to obtain three-dimensional point cloud clusters of dense parts; Step S200, a backbone neural network module is constructed on each three-dimensional point cloud of the dense parts to learn fine-grained hierarchical point high-level semantic features; a three-branch multi-task learning neural network module is constructed to detect the binary neighbor relationship and three-dimensional offset of each point relative to the center of the instance to which the point belongs, and the two-class semantic categories of each point; Step S300, the center of each part instance point cloud in the dense part cluster is calculated according to the offset, fine-grained semantic category and center neighbor relationship label of each point obtained in step S200, a three-dimensional candidate region in which each part instance is located is generated according to the average size of the same type of part, and three-dimensional instance detection of the parts in the complex machine body is realized, comprising: Step S310, for each dense part point cloud cluster, first, the instance center neighbor points and the center offset vector thereof are determined according to the center neighbor relationship label of each instance point in the cluster, and the neighbor points are collapsed to the center of the instance to which the neighbor points belong in the Euclidean space by using the center offset vector of each point; Step S320, the collapsed center neighbor points are clustered based on a breadth-first search method, the spatial coordinates of the three-dimensional neighbor points clustered into the same category are averaged to obtain the three-dimensional coordinates of the instance center to which the neighbor points belong; Step S330, according to the fine-grained semantic category of the points detected by the third branch, a three-dimensional candidate region to which each part instance belongs is defined on the original point cloud which has not been collapsed, each region takes the instance center as the region center, and three-dimensional instance detection of the parts in the complex machine body is realized by representing the parts by using the three-dimensional candidate region; Step S400, for each three-dimensional candidate region of the parts, the center features of each three-dimensional point in the region are constructed, and then the center features of each point are aggregated with the low-level geometric features of each point and the high-level semantic features learned by the backbone neural network; Step S500, a hierarchical feature learning and feature decoding neural network module is constructed to perform hierarchical encoding and decoding on the aggregated features, obtain the mask of the part instance in each three-dimensional candidate region, and regress the evaluation score of the mask, and realize instance segmentation of the dense parts in the three-dimensional point cloud of the complex aero-engine machine body.

2. The method of claim 1, wherein, The step S100 comprises: Step S110, performing semantic segmentation on the three-dimensional point cloud of the aero-engine whole machine, taking the to-be-segmented parts as foreground and the complex machine body as background, obtaining three-dimensional point clouds of various types of parts, and removing the influence of the three-dimensional point cloud of the remaining machine body of the aero-engine on the positioning and segmentation of the part instances; Step S120, performing point cloud clustering based on point density on various types of part point clouds to obtain point cloud clusters of the same type of part.

3. The method of claim 2, wherein, The step S200 comprises: Step S210, constructing a point cloud feature learning backbone neural network module to extract high-level semantic features of each point; Step S220, constructing multi-task branches, including instance center's near neighbor point detection branch, offset regression branch, and fine semantic segmentation branch.

4. The method of claim 3, wherein, In the step S220, each branch respectively performs the following tasks on each dense component point cloud cluster based on the point-level high-level semantic features learned by the backbone network: (1) detecting the binary near neighbor of each point in the dense component point cloud cluster relative to the instance center; (2) regressing the three-dimensional offset of each point relative to the instance center; (3) detecting the more fine two-class semantic category of each point.

5. The method of claim 4, wherein, The step S400 includes: for each three-dimensional candidate region of the component, constructing the center feature of each three-dimensional point in the region, and performing feature aggregation on the low-level point geometric feature and the high-level point semantic feature learned by the backbone neural network module to form the fusion feature of each point, which represents the multi-dimensional attributes of each point.

6. The method of claim 5, wherein, The step S500 includes: constructing a point cloud hierarchical feature learning and feature decoding module to perform deep feature encoding and mapping decoding on the point fusion feature of the three-dimensional candidate region, and then constructing two task branches, one of which is used to detect whether each point in the candidate region is the component instance to which the candidate region belongs, thereby obtaining the mask of the component instance; the other branch is used to regress the evaluation score of the instance mask, and finally the two branches jointly realize the instance segmentation of the dense component in the complex aero-engine body three-dimensional point cloud.

7. An aircraft engine dense part 3D instance segmentation system, comprising: The system performs the method of any one of claims 1-6.

8. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of the preceding claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of the preceding claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of the preceding claims 1-6.

Images