Open scene instance segmentation method based on decoupling distillation

Abstract

The invention provides an open scene instance segmentation method based on decoupling distillation, and the method comprises the steps: generating decoupling features of a spatial domain and a frequency domain through multi-scale feature decomposition and two-stage wavelet decomposition, calculating the semantic loss of a teacher model and a student model, carrying out the distillation training, and focusing on the alignment of the fine-grained features of the frequency domain; in order to expand a basic model to a three-dimensional scene, three-dimensional space key points are selected and projected to two dimensions to serve as prompt points, real-time segmentation is conducted through a student model, instance labels are obtained, then a mapping matrix is established with the three-dimensional key points, the instance labels are optimized through clustering, and efficient and accurate three-dimensional open scene instance segmentation is achieved. According to the method, global and local information can be fully captured, and the instance segmentation precision is improved; meanwhile, label redundancy and errors can be reduced, and the accuracy of instance labels is ensured; finally, the method has the capability of efficiently processing the three-dimensional scene, and is suitable for a quick instance segmentation task in any three-dimensional scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of instance segmentation, and in particular relates to an open scene instance segmentation method based on decoupled distillation. Background Art

[0002] Traditional instance segmentation and semantic segmentation algorithms need to be trained for given target categories. If the test samples contain new targets outside the set category set, it is difficult to achieve high-precision segmentation. With the development of large model technology, exploring zero-shot segmentation methods with open set capabilities has become a new trend. At present, large visual language models for two-dimensional images are widely used in medical image processing, defect detection and other fields due to their superior cross-scene generalization, such as SAM, CLIP, DINO, etc. However, large models for two-dimensional images cannot directly process three-dimensional open scenes, so some studies convert three-dimensional scenes into multi-view images and then map the image segmentation results to three-dimensional space.

[0003] However, the computational complexity of multi-view images is much higher than that of a single image. In order to obtain dense segmentation of three-dimensional scenes, it is necessary to use the SAM model to segment multi-view images frame by frame, but the image encoding module of the model uses a ViT network with a large number of parameters, which reduces the efficiency of the algorithm. In addition, multi-view images have problems such as target occlusion and semantic ambiguity, which will cause the same object to have very different feature distributions in images of different perspectives, reducing the accuracy of three-dimensional segmentation. Therefore, it is necessary to explore an efficient and high-precision three-dimensional scene segmentation method. The decoupled distillation proposed in the present invention can reduce the number of parameters of the model image encoding module, significantly improve the computing speed of multi-view image segmentation, and facilitate the rapid implementation of three-dimensional scene segmentation tasks.

[0004] The patent application number is 2021115338951, and the name of the invention is a Chinese patent titled "Semantic Segmentation Model Compression System and Compression Method Based on Knowledge Aggregation and Decoupling Distillation". The invention proposes a model compression method for a compression system including a feature layer comprehensive knowledge extraction and encoding module, an auxiliary predictor, and a label decoupling distillation module. The student model is optimized to make it close to the teacher's comprehensive knowledge graph, and the distillation learning of multiple semantic information and contextual knowledge is realized. Finally, the soft label distillation at the output end and the feature knowledge distillation of the intermediate layer are obtained through the label decoupling distillation module. Although this method can realize the comprehensive knowledge learning of aggregated features and enable the student model to simulate the multi-dimensional feature extraction process of the teacher model, it does not consider the extraction and learning of image detail features, which will reduce the segmentation accuracy of the student model for small samples, edges and other areas. In addition, this method only realizes semantic segmentation for closed set labels, is not suitable for open scenes, and does not have open set capabilities. Summary of the invention

[0005] To solve the above problems, the present invention provides an open scene instance segmentation method based on decoupled distillation, which can comprehensively improve the open scene segmentation efficiency and segmentation accuracy.

[0006] An open scene instance segmentation method based on decoupled distillation, comprising the following steps:

[0007] S1: Input the image I into the initial teacher model M t and the initial student model M s , we get image I in the initial teacher model M t The spatial domain decoupling feature σ under t-spatial and frequency domain decoupling characteristics σ t-freq , and in the initial student model M s The spatial domain decoupling feature σ under s-spatial and frequency domain decoupling characteristics σ s-freq ;

[0008] S2: Based on σ t-spatial , σ s-spatial Constructing spatial domain semantic loss L spatial , based on σ t-freq , σ s-freq Construct frequency domain semantic loss L freq , and use L spatial and L freq The initial teacher model M t and the initial student model M s Perform back propagation to adjust the initial teacher model M t and the initial student model M s The network parameters until L spatial and L freq are all less than the set threshold and no longer decrease with the increase of training rounds, and the student model after distillation is obtained

[0009] S3: Adopting the student model Perform instance segmentation on the multi-view two-dimensional image of the scene to be segmented, and obtain the instance label corresponding to each pixel point of the two-dimensional image;

[0010] S4: Select K key points from the three-dimensional space corresponding to the scene to be segmented using a fast histogram sampling method, and use the instance labels corresponding to the projection points of the key points on the two-dimensional image as the labels of each key point, wherein any plane with a difference between any two normal vector directions in the three-dimensional space greater than a set value has more than three key points;

[0011] S5: taking the instance labels corresponding to the key points as the instance labels of the pixels in the plane to which they belong, and obtaining a mapping matrix M between the pixels and the instance labels in the three-dimensional space;

[0012] S6: Perform instance label clustering on the mapping matrix M to obtain the instance label corresponding to each pixel point in the three-dimensional space, and complete the instance segmentation of the three-dimensional space.

[0013] Furthermore, the spatial domain decoupling feature σ t-spatial and σ s-spatial The method to obtain is:

[0014]

[0015] in, They are the initial teacher models M t Feature maps of different resolutions output by the n1th, n2th, n3th, and n4th layers of the network; They are the initial student models M s The feature maps of different resolutions output by the n1th, n2th, n3th, and n4th layers of the network; where n1, n2, n3, and n4 are the layer numbers of the four-layer network that generates the decoder input features in the initial teacher model and the initial student model, respectively, and the network layer numbers that generate the decoder input features in the two models are the same;

[0016] Frequency domain decoupling characteristic σ t-freq and σ s-freq They are composed of high-frequency features and low-frequency features, and the methods for obtaining high-frequency features and low-frequency features are:

[0017]

[0018] LL1,LH1,HL1,HH1=W2(LL)

[0019]

[0020] in, The value of and W1(·) is the first level wavelet decomposition, W2(·) is the second level wavelet decomposition, and LL represents The first-level wavelet decomposition results in the low-frequency component diagram in the horizontal direction and the low-frequency component diagram in the vertical direction. LH represents The first-level wavelet decomposition results in a low-frequency component in the horizontal direction and a high-frequency component in the vertical direction. HL represents The first-level wavelet decomposition results in a high-frequency component in the horizontal direction and a low-frequency component in the vertical direction. HH represents The high-frequency and high-frequency components in the horizontal and vertical directions obtained by the first-level wavelet decomposition are shown in Figure 1. LL1 represents the low-frequency and low-frequency components in the horizontal and vertical directions obtained by the second-level wavelet decomposition of LL. LH1 represents the low-frequency and high-frequency components in the horizontal and vertical directions obtained by the second-level wavelet decomposition of LL. HL1 represents the high-frequency and low-frequency components in the horizontal and vertical directions obtained by the second-level wavelet decomposition of LL. HH1 represents the high-frequency and high-frequency components in the horizontal and vertical directions obtained by the second-level wavelet decomposition of LL. The value of and and and are the low-frequency features of the initial teacher network and the initial student network, The value of and and and are the high-frequency features of the initial teacher network and the initial student network, respectively.

[0021] Furthermore, the spatial domain semantic loss L spatial The calculation method is:

[0022]

[0023] Among them, λ1 is the weight corresponding to the mean square distance loss MSE<·>, and λ2 is the weight corresponding to the cosine similarity loss COS<·>;

[0024] Frequency domain semantic loss L freq The calculation method is:

[0025]

[0026] Here, ||·|| represents the bi-norm.

[0027] Furthermore, the method of clustering the instance labels of the mapping matrix M to obtain the instance labels corresponding to each pixel point in the three-dimensional space is as follows:

[0028] S601: Denoise and classify each pixel in the three-dimensional space by DBSCAN clustering, and select the label with the highest probability as the cluster label corresponding to each pixel in the three-dimensional space, where the type of cluster label is the type of label remaining after the instance label is filtered out of noise points;

[0029] S602: Calculate the adjacent pixel points C that do not belong to the same cluster label respectively j and C k The intersection over union (IoU) j ,C k ), and according to the intersection over union (IoU) j ,Ck ) is not less than the set threshold θ to determine whether the two adjacent pixels belong to the same object, and complete the merging of cluster labels; if the judgment result is yes, the two adjacent pixels have the same cluster label, and the one with a higher corresponding probability is selected as the cluster label of the two adjacent pixels; if the judgment result is no, the cluster labels of the two adjacent pixels remain unchanged;

[0030] S603: The merged cluster label C merge Merge with the mapping matrix M to obtain the merged mapping matrix M updated ;

[0031] S604: For each plane in the three-dimensional space where the difference in the normal vector direction between each other is greater than the set value, calculate the mapping matrix M of all pixel points in each plane. updated The label variance ξ of the corresponding instance labels in p , the normal variance △n corresponding to all pixels, the number of all pixels N p , and then determine the ξ corresponding to each plane p ,△n,N p Whether the set threshold conditions are met at the same time, if the judgment result of any plane is yes, the instance labels of all pixels in the plane are unified as the instance label with the largest number of occurrences in the plane; if the judgment result of any plane is no, the instance labels of all pixels in the plane remain unchanged.

[0032] Furthermore, the threshold condition is set as follows:

[0033]

[0034] Among them, ε1, ε2, and ε3 are set thresholds respectively.

[0035] Beneficial effects:

[0036] 1. The present invention provides an open scene instance segmentation method based on decoupled distillation. Through multi-scale feature decomposition and two-level wavelet decomposition, decoupled features in the spatial domain and the frequency domain are generated, the semantic loss of the teacher model and the student model is calculated for distillation training, and the fine-grained features in the frequency domain are aligned. In order to expand the basic model to the three-dimensional scene, by selecting the key points in the three-dimensional space and projecting them to the two-dimensional space as the prompt points, the student model is used for real-time segmentation, and after the instance labels are obtained, a mapping matrix is ​​established with the three-dimensional key points, and the instance labels are further optimized by clustering to achieve efficient and accurate three-dimensional open scene instance segmentation. The present invention can fully capture global and local information and improve the accuracy of instance segmentation through multi-scale feature fusion and fine-grained feature alignment. At the same time, the three-dimensional key point projection and the lightweight student model are used for fast segmentation, and the label redundancy and error are reduced through clustering and optimization methods to ensure the accuracy of the instance label. Finally, the present invention has the ability to efficiently process three-dimensional scenes and is suitable for fast instance segmentation tasks in any three-dimensional scene.

[0037] 2. The present invention provides an open scene instance segmentation method based on decoupled distillation. Through multi-scale alignment of spatial features and frequency domain features, the number of parameters of the student model is significantly reduced. In addition, the present invention introduces point cloud clustering, non-maximum suppression and plane prior into the label optimization strategy, which can effectively reduce label redundancy and label error, and is suitable for any point cloud data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of decoupled distillation provided by the present invention;

[0039] Figure 2 A schematic diagram of 3D-2D label mapping provided by the present invention;

[0040] Figure 3 This is a flow chart of the label optimization strategy provided by the present invention. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0042] like Figure 1 As shown, an open scene instance segmentation method based on decoupled distillation includes the following steps:

[0043] S1: Input the image I into the initial teacher model M t and the initial student model M s , we get image I in the initial teacher model M t The spatial domain decoupling feature σ under t-spatial and frequency domain decoupling characteristics σ t-freq, and in the initial student model M s The spatial domain decoupling feature σ under s-spatial and frequency domain decoupling characteristics σ s-freq ;

[0044] It should be noted that the present invention selects two-dimensional segmentation basic models with high parameter amount SAM2_Large and low parameter amount SAM2_Tiny as teacher models M t and Student Model M s , extract the comprehensive spatial domain features of the input image I, and then perform multi-scale spatial decomposition and two-level wavelet decomposition on the features to generate decoupled features in the spatial domain and frequency domain.

[0045] Spatial domain decoupling feature σ t-spatial and σ s-spatial The method to obtain is:

[0046]

[0047] in, They are the initial teacher models M t Feature maps of different resolutions output by the n1th, n2th, n3th, and n4th layers of the network; They are the initial student models M s The feature maps of different resolutions output by the n1th, n2th, n3th, and n4th network layers; where n1, n2, n3, and n4 are the layer numbers of the four-layer network with the highest weights in the initial teacher model and the initial student model, respectively, and the network layer numbers with the highest weights in the two models are the same;

[0048] Furthermore, the frequency domain decoupling characteristic σ t-freq and σ s-freq Both are composed of high-frequency features and low-frequency features, while high-resolution feature maps It can better capture details in the image, such as edges, textures, and small objects. Wavelet decomposition can obtain rich low-frequency information and high frequency information The high-frequency information contains more image details, and the methods for obtaining high-frequency features and low-frequency features are:

[0049]

[0050] LL1,LH1,HL1,HH1=W2(LL)

[0051]

[0052] in, The value of and W1(·) is the first level wavelet decomposition, W2(·) is the second level wavelet decomposition, and LL represents The first-level wavelet decomposition results in the low-frequency component diagram in the horizontal direction and the low-frequency component diagram in the vertical direction. LH represents The first-level wavelet decomposition results in a low-frequency component in the horizontal direction and a high-frequency component in the vertical direction. HL represents The first-level wavelet decomposition results in a high-frequency component in the horizontal direction and a low-frequency component in the vertical direction. HH represents The high-frequency and high-frequency components in the horizontal and vertical directions obtained by the first-level wavelet decomposition are shown in Figure 1. LL1 represents the low-frequency and low-frequency components in the horizontal and vertical directions obtained by the second-level wavelet decomposition of LL. LH1 represents the low-frequency and high-frequency components in the horizontal and vertical directions obtained by the second-level wavelet decomposition of LL. HL1 represents the high-frequency and low-frequency components in the horizontal and vertical directions obtained by the second-level wavelet decomposition of LL. HH1 represents the high-frequency and high-frequency components in the horizontal and vertical directions obtained by the second-level wavelet decomposition of LL. The value of and and and are the low-frequency features of the initial teacher network and the initial student network, The value of and and and are the high-frequency features of the initial teacher network and the initial student network, respectively.

[0053] S2: Based on σ t-spatial , σ s-spatial Constructing spatial domain semantic loss L spatial , based on σ t-freq , σ s-freq Construct frequency domain semantic loss L freq , and use L spatial and L freq For the initial teacher model M t and the initial student model M s Perform back propagation to adjust the initial teacher model M t and the initial student model M s The network parameters until L spatial and L freq are all less than the set threshold and no longer decrease with the increase of training rounds, and the student model after distillation is obtained

[0054] Spatial domain semantic loss L spatial As long as the spatial domain decoupling feature σ t-spatial and σ s-spatialThe weighted sum of the mean square distance loss and cosine similarity loss is calculated as follows:

[0055]

[0056] Among them, λ1 is the weight corresponding to the mean square distance loss MSE<·>, and λ2 is the weight corresponding to the cosine similarity loss COS<·>;

[0057] The present invention mainly reduces the noise of high-frequency information generated by wavelet decomposition, filters out image noise, and retains key detail areas. Based on this, the frequency domain semantic loss L freq The calculation formula is as follows:

[0058]

[0059] Here, ||·|| represents the bi-norm.

[0060] It should be noted that the semantic loss of the decoupled features obtained by calculating the teacher model and the student model respectively in the present invention takes into account the global and local information and focuses on aligning the fine-grained features. Combining the two parts of the loss can achieve efficient distillation of the model.

[0061] S3: Adopting the student model Perform instance segmentation on the multi-view two-dimensional image of the scene to be segmented, and obtain the instance label corresponding to each pixel point of the two-dimensional image;

[0062] It should be noted that the distilled student model can only be used for two-dimensional instance segmentation; in order to expand to three-dimensional scenes, the present invention selects K key points from the three-dimensional space, projects the key points to the two-dimensional image as prompt points for segmentation using the internal and external parameters of the camera, and uses the prompt point label as the instance label of the key point corresponding to the prompt point, see steps S4 to S6 for details, as shown in Figure 2 shown.

[0063] S4: Use the fast histogram sampling method to select K key points from the three-dimensional space corresponding to the scene to be segmented, and use the instance labels corresponding to the projection points of the key points on the two-dimensional image as the labels of each key point, wherein any plane with a difference in the direction of any two normal vectors in the three-dimensional space greater than a set value has more than three key points, so that the edges of objects and texture detail areas with drastic changes in normal vectors will obtain more sampling points;

[0064] S5: taking the instance labels corresponding to the key points as the instance labels of the pixels in the plane to which they belong, and obtaining a Boolean mapping matrix M between the pixels and the instance labels in the three-dimensional space;

[0065] S6: Perform instance label clustering on the mapping matrix M to obtain the instance label corresponding to each pixel point in the three-dimensional space, and complete the instance segmentation of the three-dimensional space;

[0066] That is to say, after obtaining the mapping matrix M, the present invention further uses DBSCAN clustering, non-maximum suppression, and multi-plane detection to optimize instance labels and reduce label redundancy and label errors, such as Figure 3 As shown, the specific steps include:

[0067] S601: Denoising and classifying each pixel in the three-dimensional space by DBSCAN clustering, and selecting the label with the highest probability as the cluster label corresponding to each pixel in the three-dimensional space, wherein the type of the cluster label is the same as the type of the instance label;

[0068] S602: Calculate the adjacent pixel points C that do not belong to the same cluster label respectively j and C k The intersection over union (IoU) j ,C k ), and according to the intersection over union (IoU) j ,C k ) is not less than the set threshold θ to determine whether the two adjacent pixels belong to the same object, and complete the merging of cluster labels; if the judgment result is yes, the two adjacent pixels have the same cluster label, and the one with a higher corresponding probability is selected as the cluster label of the two adjacent pixels; if the judgment result is no, the cluster labels of the two adjacent pixels remain unchanged;

[0069] The above process is expressed by the following formula:

[0070] C merge =C j ∪C k If IoU(C j ,C k )≥θ

[0071] S603: The merged cluster label C merge Merge with the mapping matrix M to obtain the merged mapping matrix M updated ;

[0072] The above process is expressed by the following formula:

[0073] M updated =M∪C merge

[0074] S604: For each plane in the three-dimensional space where the difference in the normal vector direction between each other is greater than the set value, calculate the mapping matrix M of all pixel points in each plane. updated The label variance ξ of the corresponding instance labels inp , the normal variance △n corresponding to all pixels, the number of all pixels N p , and then determine the ξ corresponding to each plane p ,△n,N p Whether the set threshold conditions are met at the same time, if the judgment result of any plane is yes, the instance labels of all pixels in the plane are unified as the instance label with the largest number of occurrences in the plane; if the judgment result of any plane is no, the instance labels of all pixels in the plane remain unchanged.

[0075] The threshold conditions are:

[0076]

[0077] Among them, ε1, ε2, and ε3 are set thresholds respectively.

[0078] It can be seen that in order to solve the label redundancy problem caused by the surface texture structure, the present invention further detects the plane structure existing in the pixel points, calculates the coordinates of the pixel points in the plane and the label variance ξ p , normal difference △n and number of pixels N p Whether the threshold condition is met, the plane pixels that meet the condition are labeled and merged to reduce label redundancy.

[0079] In summary, the present invention provides an open scene instance segmentation method based on decoupled distillation, which generates decoupled features in the spatial domain and frequency domain through multi-scale feature decomposition and two-level wavelet decomposition, calculates the semantic loss of the teacher model and the student model for distillation training, and focuses on aligning the fine-grained features in the frequency domain. In order to expand the basic model to the three-dimensional scene, by selecting the key points in the three-dimensional space and projecting them to two dimensions as prompt points, the student model is used for real-time segmentation, and after obtaining the instance label, a mapping matrix is ​​established with the three-dimensional key point, and the instance label is further optimized by DBSCAN clustering, non-maximum suppression and multi-plane detection to achieve efficient and accurate three-dimensional open scene instance segmentation. This method can fully capture global and local information and improve the accuracy of instance segmentation through multi-scale feature fusion and fine-grained feature alignment. At the same time, three-dimensional key point projection and lightweight student model are used for fast segmentation, and label redundancy and errors are reduced through clustering and optimization methods to ensure the accuracy of instance labels. This method has the ability to efficiently process three-dimensional scenes and is suitable for fast instance segmentation tasks in any three-dimensional scene.

[0080] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may certainly make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. An open scene instance segmentation method based on decoupled distillation, characterized in that: The following steps are involved: S1: Input the image I into the initial teacher model M t and the initial student model M s , we get image I in the initial teacher model M t The spatial domain decoupling feature σ under t-spatial and frequency domain decoupling characteristic σ t-freq , and in the initial student model M s The spatial domain decoupling feature σ under s-spatial and frequency domain decoupling characteristic σ s-freq ; S2: Based on σ t-spatial , σ s-spatial Constructing spatial domain semantic loss L spatial , based on σ t-freq , σ s-freq Construct frequency domain semantic loss L freq , and use L spatial and L freq The initial teacher model M t and the initial student model M s Perform back propagation to adjust the initial teacher model M t and the initial student model M s The network parameters until L spatial and L freq are all less than the set threshold and no longer decrease with the increase of training rounds, and the student model after distillation is obtained S3: Adopting the student model Perform instance segmentation on the multi-view two-dimensional image of the scene to be segmented, and obtain the instance label corresponding to each pixel point of the two-dimensional image; S4: Select K key points from the three-dimensional space corresponding to the scene to be segmented using a fast histogram sampling method, and use the instance labels corresponding to the projection points of the key points on the two-dimensional image as the labels of each key point, wherein any plane with a difference between any two normal vector directions in the three-dimensional space greater than a set value has more than three key points; S5: taking the instance labels corresponding to the key points as the instance labels of the pixels in the plane to which they belong, and obtaining a mapping matrix M between the pixels and the instance labels in the three-dimensional space; S6: Perform instance label clustering on the mapping matrix M to obtain the instance label corresponding to each pixel point in the three-dimensional space, and complete the instance segmentation of the three-dimensional space.

2. The open scene instance segmentation method based on decoupled distillation according to claim 1, characterized in that: Spatial domain decoupling feature σ t-spatial and σ s-spatial The method to obtain is: in, They are the initial teacher models M t Feature maps of different resolutions output by the n1th, n2th, n3th, and n4th layers of the network; They are the initial student models M s The feature maps of different resolutions output by the n1th, n2th, n3th, and n4th layers of the network; where n1, n2, n3, and n4 are the layer numbers of the four-layer network that generates the decoder input features in the initial teacher model and the initial student model, respectively, and the network layer numbers that generate the decoder input features in the two models are the same; Frequency domain decoupling characteristic σ t-freq and σ s-freq They are composed of high-frequency features and low-frequency features, and the methods for obtaining high-frequency features and low-frequency features are: LL1,LH1,HL1,HH1=W2(LL) in, The value of and W1(·) is the first level wavelet decomposition, W2(·) is the second level wavelet decomposition, and LL represents The first-level wavelet decomposition results in the low-frequency component diagram in the horizontal direction and the low-frequency component diagram in the vertical direction. LH represents The first-level wavelet decomposition results in a low-frequency component in the horizontal direction and a high-frequency component in the vertical direction. HL represents The first-level wavelet decomposition results in a high-frequency component in the horizontal direction and a low-frequency component in the vertical direction. HH represents The high-frequency and high-frequency components in the horizontal and vertical directions obtained by the first-level wavelet decomposition are shown in Figure 1. LL1 represents the low-frequency and low-frequency components in the horizontal and vertical directions obtained by the second-level wavelet decomposition of LL. LH1 represents the low-frequency and high-frequency components in the horizontal and vertical directions obtained by the second-level wavelet decomposition of LL. HL1 represents the high-frequency and low-frequency components in the horizontal and vertical directions obtained by the second-level wavelet decomposition of LL. HH1 represents the high-frequency and high-frequency components in the horizontal and vertical directions obtained by the second-level wavelet decomposition of LL. The value of and and and are the low-frequency features of the initial teacher network and the initial student network, The value of and and and are the high-frequency features of the initial teacher network and the initial student network, respectively.

3. The open scene instance segmentation method based on decoupled distillation according to claim 2, characterized in that: Spatial domain semantic loss L spatial The calculation method is: Among them, λ1 is the weight corresponding to the mean square distance loss MSE<·>, and λ2 is the weight corresponding to the cosine similarity loss COS<·>; Frequency domain semantic loss L freq The calculation method is: Here, ||·|| represents the bi-norm.

4. The open scene instance segmentation method based on decoupled distillation according to claim 1, characterized in that: The method of clustering the instance labels of the mapping matrix M to obtain the instance labels corresponding to each pixel in the three-dimensional space is as follows: S601: Denoise and classify each pixel in the three-dimensional space by DBSCAN clustering, and select the label with the highest probability as the cluster label corresponding to each pixel in the three-dimensional space, where the type of cluster label is the type of label remaining after the instance label is filtered out of noise points; S602: Calculate the adjacent pixel points C that do not belong to the same cluster label respectively j and C k The intersection over union (IoU) j ,C k ), and according to the intersection over union (IoU) j ,C k ) is not less than the set threshold θ to determine whether the two adjacent pixels belong to the same object, and complete the merging of cluster labels; if the judgment result is yes, the two adjacent pixels have the same cluster label, and the one with a higher corresponding probability is selected as the cluster label of the two adjacent pixels; if the judgment result is no, the cluster labels of the two adjacent pixels remain unchanged; S603: The merged cluster label C merge Merge with the mapping matrix M to obtain the merged mapping matrix M updated ; S604: For each plane in the three-dimensional space where the difference in the normal vector direction between each other is greater than the set value, calculate the mapping matrix M of all pixel points in each plane. updated The label variance ξ of the corresponding instance labels in p , the normal variance △n corresponding to all pixels, the number of all pixels N p , and then determine the ξ corresponding to each plane p ,△n,N p Whether the set threshold conditions are met at the same time, if the judgment result of any plane is yes, the instance labels of all pixels in the plane are unified as the instance label with the largest number of occurrences in the plane; if the judgment result of any plane is no, the instance labels of all pixels in the plane remain unchanged.

5. The open scene instance segmentation method based on decoupled distillation according to claim 4, characterized in that: The threshold conditions are: Among them, ε1, ε2, and ε3 are set thresholds respectively.

Images