A monocular 3D instance segmentation method guided by depth information

Abstract

The present invention proposes a monocular three-dimensional instance segmentation method based on depth information guidance. The method includes an instance information and depth information acquisition module and a three-dimensional instance segmentation module. The instance information and depth information acquisition module adopts a multi-task network to perform two-dimensional instance segmentation and depth estimation at the same time, wherein, when performing instance segmentation, the depth features in the depth estimation branch are introduced to further improve the accuracy of instance segmentation. The three-dimensional instance segmentation module uses the acquired instance information and depth information to generate a three-dimensional point cloud with instance information, and then designs an adaptive point cloud filtering unit to filter the noise points in the point cloud to achieve monocular three-dimensional instance segmentation of the target. The beneficial effect of the present invention is that the monocular three-dimensional instance segmentation method based on depth information guidance can accurately perform three-dimensional instance segmentation of scene targets in application scenarios where there is only an RGB camera and lacks 3D instance segmentation data labels.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning, and specifically relates to a monocular three-dimensional instance segmentation method based on depth information guidance. Background Art

[0002] With the development of the industrial Internet, production workshops are gradually becoming intelligent, and one of the primary prerequisites for workshop intelligence is the accurate identification of workshop scene targets. As an effective method for scene target recognition, three-dimensional instance segmentation can not only obtain the category information of the target, but also obtain the three-dimensional spatial position information that two-dimensional instance segmentation cannot obtain, which significantly improves the accuracy of basic applications such as motion estimation and distance measurement. Because of these characteristics, three-dimensional instance segmentation has been widely used in fields such as autonomous driving, virtual reality enhancement, and robot navigation. However, most of the current target recognition methods in workshop scenes are based on two-dimensional information, and few three-dimensional information is applied to workshop scenes. If three-dimensional instance segmentation is introduced into the production workshop, it will effectively improve the accuracy of applications such as workshop safety monitoring and workshop AGV navigation and obstacle avoidance. Therefore, it is very necessary to study three-dimensional instance segmentation in the workshop.

[0003] However, due to the particularity and complexity of the smart workshop scene, the recognition of smart workshop scene targets faces the following two difficulties:

[0004] (1) In current production workshops, most information acquisition devices are RGB monocular cameras, which can only obtain two-dimensional spatial information but not three-dimensional information. Therefore, how to achieve accurate three-dimensional instance segmentation with only two-dimensional spatial information is a major challenge.

[0005] (2) Due to the particularity of the production workshop scene, data samples in the workshop scene are difficult to collect and the labeling cost is extremely high. Therefore, it is a difficult problem to achieve accurate 3D instance segmentation in the absence of 3D instance segmentation labels.

[0006] In summary, to address the above difficulties, a monocular 3D instance segmentation method guided by depth information is proposed. With only a monocular RGB camera and no 3D instance segmentation labels required, accurate 3D instance segmentation of production workshop scene targets is achieved.

[0007] Invention Content

[0008] In view of the above problems, the purpose of the present invention is to provide a monocular 3D instance segmentation method based on depth information guidance. The method realizes depth estimation while identifying scene targets by improving the Deep-snake network model, and realizes efficient 3D instance segmentation by relying only on a monocular RGB camera without the need for 3D instance segmentation labels.

[0009] A monocular 3D instance segmentation method based on depth information guidance comprises the following steps:

[0010] S1. Establishing a 3D instance segmentation model: The 3D instance segmentation model includes an instance information and depth information acquisition module and a 3D instance segmentation module;

[0011] The instance information and depth information acquisition module, i.e., the multi-task network, includes an encoding module and a decoding module; the encoding module is composed of a DLA34 backbone network, and is used to extract features from the input color image to obtain feature maps of different sizes; the decoding module uses the feature maps of different sizes generated by the encoding module to simultaneously perform instance segmentation and depth estimation to obtain a two-dimensional instance mask and a depth image;

[0012] The three-dimensional instance segmentation module includes a point cloud generation unit and an adaptive point cloud filtering unit; the point cloud generation unit generates a three-dimensional point cloud with instance information using the two-dimensional instance mask and the depth image generated by the instance information and depth information acquisition module; the adaptive point cloud filtering unit performs adaptive point cloud filtering processing on the three-dimensional point cloud with instance information generated by the point cloud generation unit to obtain an accurate three-dimensional instance segmentation result;

[0013] S2. Training the multi-task network: inputting the color and depth maps and their corresponding two-dimensional instance segmentation labels into the multi-task network, performing end-to-end training, and obtaining a trained multi-task network;

[0014] S3. Monocular 3D instance segmentation: The color image is input into the trained multi-task network to obtain a 2D instance mask and a depth image. The 2D instance mask and the depth image are then input into the 3D instance segmentation module to generate the final 3D instance segmentation result.

[0015] Further, the decoding module includes a depth estimation branch and a two-dimensional instance segmentation branch;

[0016] The depth estimation branch uses a deconvolution operation to upsample the lower layer features in the feature maps of different sizes generated in the encoding module to adjust the feature size, and then performs a convolution operation on the upper layer features to adjust the number of channels, and performs an element-wise fusion process on the adjusted two features to generate fused features, and iterates three times to obtain three-layer fused features, and finally performs a deconvolution operation on the top layer features again to generate a depth image of a size compared to the input color image, and in addition, the top layer features are passed as depth features to the two-dimensional instance segmentation branch to assist in predicting the two-dimensional instance mask;

[0017] The two-dimensional instance segmentation branch uses the 0th layer features in the feature maps of different sizes generated by the encoding module and the depth features passed in by the depth estimation branch to jointly determine the boundary of the target and generate a two-dimensional instance mask.

[0018] Furthermore, the point cloud generation unit uses the multi-task network to generate two-dimensional instance information, namely, instance mask, to crop the predicted depth image, retain the depth information at the position corresponding to the mask, and uses the following formula to perform point cloud projection to generate a three-dimensional point cloud with instance information:

[0019]

[0020] Where (x, y, z) is the spatial coordinate of the three-dimensional point cloud, (x ′ ,y ′ ) is the pixel coordinate in the depth map, D is the depth value at the corresponding position in the depth map, and f x , f y are the focal lengths of the lens in the x and y directions respectively.

[0021] Furthermore, the statistical filtering algorithm combining color information and distance information first traverses the point cloud P filtered by the adaptive through-filtering algorithm. ′ For all points in , determine the neighboring points of each point, where the neighboring points are the n2 points centered on the point and closest to the point, calculate the average distance and color difference between the point and the n2 neighboring points; use the average distance and color difference of each point to calculate the average distance and color difference of the entire point cloud. With standard deviation The color threshold is determined by using the average and standard deviation of the distance and color difference of the entire point cloud. color Distance threshold dis ; Traverse the point cloud again and use the average distance and color difference of each point to obtain the color deviation score of each point Deviation score from distance And use the weighted sum method to obtain the joint deviation score of each point Finally, the joint deviation score is used to identify and filter the point cloud noise.

[0022]

[0023]

[0024]

[0025]

[0026]

[0027]

[0028] In the formula, Respectively represent the average color difference and average distance of the entire point cloud, Respectively represent the color difference standard deviation and distance standard deviation of the entire point cloud, ratio s Represents the weight coefficient, which is used to measure the influence of the mean and standard deviation. They represent the color deviation score and distance deviation score of point a respectively. Respectively represent the average color difference and average distance between the a-th point and its neighboring points, Weight dis ,Weight color Respectively represent the distance score weight and color score weight hyperparameters, Represents the joint deviation score of the current point. Res is the judgment result of the current point. When Res is filter, it means that the current point is classified as a noise point and needs to be filtered. When Res is retain, it means that the current point is classified as a target point and is retained.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. It is possible to obtain target instance information and three-dimensional space information with only an RGB camera.

[0031] 2. Achieve accurate 3D instance segmentation with a small amount of 2D annotation data and no 3D annotation data.

[0032] 3. The monocular 3D instance segmentation method guided by depth information meets the real-time and accuracy requirements of 3D instance segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is the overall flow chart of the monocular 3D instance segmentation method guided by depth information.

[0034] Figure 2 It is the overall flow chart of the multi-task network.

[0035] Figure 3 It is the overall flow chart of 3D instance segmentation.

[0036] Figure 4 It is the depth estimation branch.

[0037] Figure 5 is the instance segmentation branch.

[0038] Figure 6 It is the offset acquisition module.

[0039] Figure 7 It is the bias weight acquisition unit.

[0040] Figure 8 is the attention module.

[0041] Fig. 9 This is the final effect picture. DETAILED DESCRIPTION

[0042] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0043] A monocular 3D instance segmentation method based on depth information guidance, taking a digital workshop as an example, specifically includes the following steps:

[0044] S1. Establish a 3D instance segmentation model: as shown in the attached Figure 1 As shown, the 3D instance segmentation model includes an instance information and depth information acquisition module and a 3D instance segmentation module;

[0045] Among them, the instance information and depth information acquisition module, namely the multi-task network, is shown in the attached Figure 2 As shown, it includes an encoding module and a decoding module; the encoding module is composed of a DLA34 backbone network, which is used to extract features from the input color image to obtain feature maps of different sizes; the decoding module uses the feature maps of different sizes generated by the encoding module to simultaneously perform instance segmentation and depth estimation to obtain a two-dimensional instance mask and a depth image;

[0046] Among them, the 3D instance segmentation module, as shown in the attached Figure 3 As shown, it includes a point cloud generation unit and an adaptive point cloud filtering unit; the point cloud generation unit generates a three-dimensional point cloud with instance information using the two-dimensional instance mask and the depth image generated by the instance information and depth information acquisition module; the adaptive point cloud filtering unit performs adaptive point cloud filtering processing on the three-dimensional point cloud with instance information generated by the point cloud generation unit to obtain an accurate three-dimensional instance segmentation result;

[0047] The decoding module includes a depth estimation branch and a two-dimensional instance segmentation branch;

[0048] Among them, as attached Figure 4As shown, the depth estimation branch first performs a deconvolution operation on the third layer features in the feature maps of different sizes generated by the encoding module to adjust the feature size, and then uses a convolution operation to adjust the number of feature channels on the second layer features. Finally, the adjusted two layers of features are merged to generate a fused second layer feature. This is iterated three times to generate fused 0th, 1st, and 2nd layer features. Finally, the fused 0th layer features are deconvolved again to generate a depth image of the size of the original color image. In addition, the fused 0th layer features are passed as depth features to the two-dimensional instance segmentation branch to assist in predicting the two-dimensional instance mask.

[0049] The two-dimensional instance segmentation branch uses the 0th layer features in the feature maps of different sizes generated by the encoding module and the depth features passed in by the depth estimation branch to jointly determine the boundary of the target and generate a two-dimensional instance mask;

[0050] Among them, as attached Figure 5 As shown, the above-mentioned two-dimensional instance segmentation branch includes an initial boundary generation module, a boundary offset acquisition module, and a boundary deformation module, wherein the two-dimensional instance segmentation branch uses the initial boundary generation module to generate the target initial boundary, and iteratively executes the boundary offset acquisition module and the boundary deformation model to deform the initial boundary, and generates a predicted boundary to fit the real target boundary;

[0051] The initial boundary generation module uses the feature maps of different sizes generated by the encoding module to perform feature upsampling processing and target detection frame prediction to generate features and target detection frames containing high-level semantic information, that is, the target initial boundary;

[0052] The boundary offset acquisition module uses the features containing high-level semantic information generated by the initial boundary generation module and the depth features passed in by the depth estimation branch to jointly determine the boundary offset for adjusting the initial boundary.

[0053] The boundary deformation module uses the boundary offset generated by the boundary offset acquisition module to perform an element-by-element addition adjustment operation on the initial boundary to fit the real target boundary.

[0054] Among them, as attached Figure 6 As shown, the boundary offset acquisition module includes two offset prediction networks and an offset weight acquisition unit; the boundary offset acquisition module uses the offset weight generated by the offset weight acquisition unit to perform weighted summation on the offsets predicted by the two offset prediction networks to generate a boundary offset. The operation in the above boundary deformation module is expressed by the following formula:

[0055]

[0056] Where, offset AllIndicates the boundary offset, Respectively represent the offset weights of the two modal features, offset RGB , offset Depth Respectively represent the offsets of the two modes;

[0057] The two offset prediction networks are two sub-networks with the same structure, which are respectively used to predict the offsets of the color and depth modal features; the sub-networks include a feature extraction module, a feature fusion module and a prediction module; the feature extraction module further extracts features containing high-level semantic information and depth features generated by the initial boundary generation module to obtain offset features; the feature fusion module fuses the obtained offset features to obtain fused features; the prediction module predicts the offset using the fused features;

[0058] The offset weight acquisition unit generates offset weights of the two modal features respectively using the fused features generated by the feature fusion module.

[0059] Among them, as attached Figure 7 As shown, the above-mentioned offset weight acquisition unit first splices and fuses the fused features of the two modalities generated by the feature fusion module according to the channel dimension to generate spliced ​​features; the spliced ​​features are passed into the attention module to obtain the initial weights of the two modalities; the initial weights of the two modalities are spliced ​​and fused according to the channel dimension, and the Softmax operation is performed to normalize the weights to obtain the normalized offset weight vector; finally, the normalized offset weight vector is split to obtain the offset weights of the color modality and the depth modality: The operation in the above-mentioned offset weight acquisition unit is expressed by the following formula:

[0060] Weight RGB =WA RGB (Concat(RGB in , Depth in ))

[0061] Weight Depth =WA Depth (Concat(RGB in , Depth in ))

[0062] Weight All =Softmax(Concat(Weight RGB ,Weight Detph ))

[0063]

[0064]

[0065] In the formula, RGB in , Depth in They represent the fused features of the color mode and the depth mode respectively. Concat represents feature concatenation according to the channel dimension. WA RGB , W.A. Depth Respectively represent the attention modules of the two modalities, Weight RGB ,Weight Depth represents the initial weights of the color mode and the depth mode, Softmax represents the normalization operation of the feature, Split RGB , Split Depth Respectively represent the splitting operation of the vector, which obtains the first half and the second half of the vector respectively, Weight All represents the normalized weight vector, Respectively represent the offset weights of the two modes;

[0066] Among them, as attached Figure 8 As shown, the attention module first performs a one-dimensional convolution operation on the concatenated features to compress the number of channels, and then passes the features into the linear layer, the Relu activation layer, and the linear layer to obtain the initial weights. The operations in the above attention module can be expressed by the following formula:

[0067] Weight = LRL (Conv (feature merged ))

[0068] In the formula, feattre merged Represents the concatenated features, Conv represents a one-dimensional convolution operation, LRL represents passing the features into the linear layer, Relu activation layer, and linear layer for processing, and Weight represents the initial weight obtained;

[0069] The point cloud generation unit uses the multi-task network to generate two-dimensional instance information, namely, instance mask, to crop the predicted depth image, retain the depth information at the position corresponding to the mask, set the depth value of the remaining area to 0, and use the following formula to perform point cloud projection to generate a three-dimensional point cloud with instance information:

[0070]

[0071] Where (x, y, z) is the spatial coordinate of the 3D point cloud, (x', y') is the pixel coordinate in the depth map, D is the depth value at the corresponding position in the depth map, and f x , f yare the focal lengths of the lens in the x and y directions respectively.

[0072] The above-mentioned adaptive point cloud filtering unit includes an adaptive through-filtering algorithm and a statistical filtering algorithm combining color information and distance information;

[0073] The adaptive through-filter algorithm first calculates the mean value z of the coordinates on the Z axis of the three-dimensional point cloud with instance information generated by the point cloud generation unit. mean With standard deviation z std , and then use the obtained mean and standard deviation to jointly determine the distance threshold of the adaptive through-filter to achieve accurate filtering of the background noise in the point cloud. The operations in the above adaptive through-filter algorithm can be expressed by the following formula:

[0074] p′ i (x i ,y i , z i )∈P',st.((z mean -ratio p × std )<z i <(z mean +ratio p × std )), i = {1, 2, ..., n1}

[0075]

[0076]

[0077] In the formula, ratio p Represents the weight coefficient, which is used to balance the standard deviation and the mean. n is the number of points in the point cloud before filtering, n1 is the number of points in the point cloud after filtering, and z i is the projection coordinate of each point in the point cloud set on the z-axis, and P' represents the point cloud filtered by the adaptive through-filtering algorithm;

[0078] The statistical filtering algorithm combining color information and distance information first traverses all points in the point cloud P' filtered by the adaptive straight-through filtering algorithm, determines the neighboring points of each point, and the neighboring points are n2 points centered on the point and closest to the point, and calculates the average distance and color difference between the point and the n2 neighboring points; the average distance and color difference of each point are used to calculate the average value of the distance and color difference of the entire point cloud. With standard deviation The color threshold is determined by using the average and standard deviation of the distance and color difference of the entire point cloud. color Distance threshold dis; Traverse the point cloud again and use the average distance and color difference of each point to obtain the color deviation score of each point Deviation score from distance And use the weighted sum method to obtain the joint deviation score of each point Finally, the point cloud noise is identified and filtered by the joint deviation score. The operation in the statistical filtering algorithm combining color information and distance information can be expressed by the following formula:

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091] Where n2 is the number of neighbor points, n1 is the number of points in the point cloud after filtering by the adaptive through-filtering algorithm, (R j ,G j ,B j ) represents the color intensity value at the jth point, (X j ,Y j ,Z j ) represents the spatial position coordinates of the jth point, Respectively represent the average color difference and distance between the jth point and its neighboring points, Respectively represent the color difference mean and distance average of the entire point cloud, Respectively represent the color difference standard deviation and distance standard deviation of the entire point cloud, ratio s Represents the weight coefficient, which is used to measure the influence of the mean and standard deviation. They represent the color deviation score and distance deviation score of point a respectively. Respectively represent the average color difference and average distance between the a-th point and its neighboring points, Weight dis ,Weight color Respectively represent the distance score weight and color score weight hyperparameters, Represents the joint deviation score of the current point. Res is the judgment result of the current point. When Res is filter, it means that the current point is classified as a noise point and needs to be filtered. When Res is retain, it means that the current point is classified as a target point and is retained.

[0092] S2. Training the multi-task network: inputting the color and depth maps and their corresponding two-dimensional instance segmentation labels into the multi-task network, performing end-to-end training, and obtaining a trained multi-task network;

[0093] S3. Monocular 3D instance segmentation: Input the color image into the trained multi-task network to obtain the 2D instance mask and depth image. Then input the 2D instance mask and depth image into the 3D instance segmentation module to generate the final 3D instance segmentation result, as shown in the attached figure. Fig. 9 shown.

[0094] The entire monocular 3D instance segmentation is fully described as follows:

[0095] Step 1: Set the color image resolution of the entire network input to 480×460 pixels;

[0096] Step 2: Input the color image into the encoding module for feature extraction to obtain feature maps of different sizes;

[0097] Step 3: The feature maps of different sizes are passed into the decoding module to perform two-dimensional instance segmentation and depth estimation, and obtain the two-dimensional instance mask and depth image; the depth estimation branch uses the obtained feature maps of different sizes to predict the depth value, and passes the depth features into the two-dimensional instance segmentation branch. The two-dimensional instance segmentation branch first passes the features passed into the encoding module into its boundary generation module to generate the initial boundary of the target and the features containing high-level semantic information. Then, the boundary offset acquisition module uses the depth features and the features containing high-level semantic information to jointly determine the boundary offset, and passes the boundary offset into the boundary deformation module to adjust the initial boundary of the target to achieve the fitting of the target boundary. Among them, the two offset prediction networks in the boundary offset module first use their respective feature extraction modules to extract the features passed in, and pass them into the feature fusion module for feature fusion. Then, the prediction module uses the fused features to predict the offset; the boundary offset module uses the offset weights obtained by its offset weight acquisition module to adjust the offset to obtain the boundary offset.

[0098] Step 4: The point cloud generation unit first crops the depth image using the predicted 2D instance mask and generates a 3D point cloud with instance information using coordinate projection transformation.

[0099] Step 5: The adaptive point cloud filtering unit first uses adaptive pass-through filtering to filter the background noise in the point cloud; then, it combines the statistical filtering of color information and depth information to filter the discrete noise around the target to complete the final monocular 3D instance segmentation.

Claims

1. A monocular 3D instance segmentation method based on depth information guidance, characterized in that: The steps include: S1. Establishing a 3D instance segmentation model: The 3D instance segmentation model includes an instance information and depth information acquisition module and a 3D instance segmentation module; The instance information and depth information acquisition module, i.e., the multi-task network, includes an encoding module and a decoding module; the encoding module is composed of a DLA34 backbone network, and is used to extract features from the input color image to obtain feature maps of different sizes; the decoding module uses the feature maps of different sizes generated by the encoding module to perform instance segmentation and depth estimation, and obtain a two-dimensional instance mask and a depth image; The three-dimensional instance segmentation module includes a point cloud generation unit and an adaptive point cloud filtering unit; the point cloud generation unit generates a three-dimensional point cloud with instance information using the two-dimensional instance mask and the depth image generated by the instance information and depth information acquisition module; the adaptive point cloud filtering unit performs adaptive point cloud filtering processing on the three-dimensional point cloud with instance information generated by the point cloud generation unit to obtain an accurate three-dimensional instance segmentation result; S2. Training the multi-task network: inputting the color and depth maps and their corresponding two-dimensional instance segmentation labels into the multi-task network, performing end-to-end training, and obtaining a trained multi-task network; S3. Monocular 3D instance segmentation: The color image is input into the trained multi-task network to obtain a 2D instance mask and a depth image. The 2D instance mask and the depth image are then input into the 3D instance segmentation module to generate the final 3D instance segmentation result.

2. The method for monocular 3D instance segmentation based on depth information guidance according to claim 1, characterized in that: The decoding module includes a depth estimation branch and a two-dimensional instance segmentation branch; The depth estimation branch uses a deconvolution operation to upsample the lower layer features in the feature maps of different sizes generated in the encoding module to adjust the feature size, and then performs a convolution operation on the upper layer features to adjust the number of channels, and performs an element-wise fusion process on the adjusted two features to generate fused features, and iterates three times to obtain three-layer fused features, and finally performs a deconvolution operation on the top layer features again to generate a depth image of a size compared to the input color image, and in addition, the top layer features are passed as depth features to the two-dimensional instance segmentation branch to assist in predicting the two-dimensional instance mask; The two-dimensional instance segmentation branch uses the 0th layer features in the feature maps of different sizes generated by the encoding module and the depth features passed in by the depth estimation branch to jointly determine the boundary of the target and generate a two-dimensional instance mask.

3. The method for monocular 3D instance segmentation based on depth information guidance according to claim 2, characterized in that: The two-dimensional instance segmentation branch includes an initial boundary generation module, a boundary offset acquisition module, and a boundary deformation module, wherein the two-dimensional instance segmentation branch uses the initial boundary generation module to generate an initial boundary of the target, and iteratively executes the boundary offset acquisition module and the boundary deformation model to deform the initial boundary to generate a predicted boundary to fit the real target boundary; The initial boundary generation module uses the feature maps of different sizes generated by the encoding module to perform feature upsampling processing and target detection frame prediction to generate features and target detection frames containing high-level semantic information, that is, the target initial boundary; The boundary offset acquisition module uses the features containing high-level semantic information generated by the initial boundary generation module and the depth features passed in by the depth estimation branch to jointly determine the boundary offset for adjusting the initial boundary; The boundary deformation module uses the boundary offset generated by the boundary offset acquisition module to perform an element-by-element addition adjustment operation on the initial boundary to fit the real target boundary.

4. The method for monocular 3D instance segmentation based on depth information guidance according to claim 3, characterized in that: The boundary offset acquisition module includes two offset prediction networks and an offset weight acquisition unit; the offsets predicted by the two offset prediction networks are weighted summed using the offset weights generated by the offset weight acquisition unit to generate the boundary offset. The operation in the above boundary deformation module is expressed by the following formula: Where, offset All Indicates the boundary offset, Respectively represent the offset weights of the two modal features, offset RGB , offset Depth Respectively represent the offsets of the two modes; The two offset prediction networks are two sub-networks with the same structure, which are respectively used to predict the offsets of the color and depth modal features; the sub-networks include a feature extraction module, a feature fusion module and a prediction module; the feature extraction module further extracts features containing high-level semantic information and depth features generated by the initial boundary generation module to obtain offset features; the feature fusion module fuses the obtained offset features to obtain fused features; the prediction module predicts the offset using the fused features; The offset weight acquisition unit generates offset weights of the two modal features respectively using the fused features generated by the feature fusion module.

5. The method for monocular 3D instance segmentation based on depth information guidance according to claim 4, characterized in that: The offset weight acquisition unit first concatenates and fuses the fused features of the two modalities generated by the feature fusion module according to the channel dimension to generate concatenated features; The concatenated features are passed into the attention module to obtain the initial weights of the two modalities. The initial weights of the two modalities are concatenated and fused according to the channel dimension, and the weights are normalized by the Softmax operation to obtain the normalized offset weight vector. Finally, the normalized offset weight vector is split to obtain the offset weights of the color modality and the depth modality. The operation in the above offset weight acquisition unit is expressed by the following formula: Weight RGB =WA RGB (Concat(RGB in ,Depth in )) Weight Depth =WA Depth (Concat(RGB in ,Depth in )) Weight All =Softmax(Concat(Weight RGB ,Weight Detph )) In the formula, RGB in , Depth in They represent the fused features of the color mode and the depth mode respectively. Concat represents feature concatenation according to the channel dimension. WA RGB , W.A. Depth Respectively represent the attention modules of the two modalities, Weight RGB ,Weight Depth represents the initial weights of the color mode and the depth mode, Softmax represents the normalization operation of the feature, Split RGB , Split Depth Respectively represent the splitting operation of the vector, which obtains the first half and the second half of the vector respectively, Weight All represents the normalized weight vector, Respectively represent the offset weights of the two modes; The attention module first performs a one-dimensional convolution operation on the concatenated features to compress the number of channels, and then passes the features into the linear layer, activation layer, and linear layer in sequence to obtain the initial weights. The operations in the above attention module can be expressed by the following formula: Weight=LRL(Conv(feature merged )) In the formula, feature merged Represents the concatenated features, Conv represents the one-dimensional convolution operation, LRL represents the features are passed to the linear layer, Relu activation layer, and linear layer in sequence for processing, and Weight represents the initial weight obtained.

6. The method for monocular 3D instance segmentation based on depth information guidance according to claim 1, characterized in that: The adaptive point cloud filtering unit includes an adaptive through-filtering algorithm and a statistical filtering algorithm combining color information and distance information; The adaptive through-filter algorithm first calculates the mean value z of the coordinates on the Z axis of the three-dimensional point cloud with instance information generated by the point cloud generation unit. mean With standard deviation z std , and then use the obtained mean and standard deviation to jointly determine the distance threshold of the adaptive through-filter to achieve accurate filtering of the background noise in the point cloud. The operations in the above adaptive through-filter algorithm can be expressed by the following formula: p′ i (x i ,y i ,z i )∈P’,st.((z mean -ratio p ×z std )<z i <(z mean +ratio p ×z std )), i={1,2,...,n1} In the formula, ratio p represents the weight coefficient, which is used to balance the standard deviation and the mean. n is the number of points in the point cloud before filtering, n1 is the number of points in the point cloud after filtering, and z i is the projection coordinate of the i-th point in the point cloud on the z-axis, and P′ represents the point cloud filtered by the adaptive through-filtering algorithm; The statistical filtering algorithm combining color information and distance information first traverses all points in the point cloud P′ filtered by the adaptive straight-through filtering algorithm, determines the neighboring points of each point, and the neighboring points are n2 points centered on the point and closest to the point, and calculates the average distance and color difference between the point and the n2 neighboring points; the average distance and color difference of each point are used to calculate the average value of the distance and color difference of the entire point cloud. With standard deviation The color threshold is determined by using the average and standard deviation of the distance and color difference of the entire point cloud. color Distance threshold dis ; Traverse the point cloud again, use the average distance and color difference of each point to obtain the color deviation score and distance deviation score of each point, and use the weighted summation method to obtain the joint deviation score of each point. Finally, the joint deviation score is used to identify and filter the point cloud noise. The above operation in the statistical filtering algorithm combining color information and distance information can be expressed by the following formula: Where n2 is the number of neighbor points, n1 is the number of points in the point cloud after filtering by the adaptive through-filtering algorithm, (R j , G j , B j ) represents the color intensity value at the jth point, (X j , Y j , Z j ) represents the spatial position coordinates of the jth point, Respectively represent the average color difference and distance between the jth point and its neighboring points, Respectively represent the average color difference and average distance of the entire point cloud, Respectively represent the color difference standard deviation and distance standard deviation of the entire point cloud, ratio s Represents the weight coefficient, which is used to measure the influence of the mean and standard deviation. They represent the color deviation score and distance deviation score of point a respectively. Respectively represent the average color difference and average distance between the a-th point and its neighboring points, Weight dis ,Weight color Respectively represent the distance score weight and color score weight hyperparameters, Represents the joint deviation score of the current point. Res is the judgment result of the current point. When Res is filter, it means that the current point is classified as a noise point and needs to be filtered. When Res is retain, it means that the current point is classified as a target point and is retained.

Images