Instance segmentation method, device and detection system based on RGBD image

Abstract

The present application relates to an instance segmentation method and device based on RGBD images, including: obtaining the current RGB image and actual depth information of the current specific scene; outputting feature maps of four different stages after convolution operation based on the residual network of the convolutional neural network; obtaining the category probability that each point on the feature map belongs to a certain category center point and a dynamic convolution kernel that can represent the instance; and obtaining a segmentation mask of the object corresponding to the point, selecting the category probability greater than the second reference threshold as the final output result, thereby achieving accurate instance segmentation. The present invention innovatively uses the depth information twice, one is to gradually merge it with the RGB information as auxiliary information of RGB in the feature extraction stage; the second is to enhance the spatial position information contained in the feature map through dynamic depth coordinate convolution. Therefore, compared with the traditional segmentation model based only on RGB, more accurate instance segmentation can be achieved.

Description

Technical Field

[0001] The present application relates to the field of computer image vision processing, and in particular to an instance segmentation method and device, and a detection system based on RGBD images. Background Art

[0002] Instance segmentation is a type of image segmentation, which is a combination of semantic segmentation and object detection. Instance segmentation requires not only finding the precise object coordinates, but also finding the accurate instance mask.

[0003] With the rapid development of depth sensors such as Kinect, it has become cheaper to obtain depth information of scenes. Depth information can be used as a supplement to RGB information, so as to better segment objects that are difficult to segment only by relying on RGB images, such as close objects with different depths but similar textures and colors. There are other instance segmentation methods on the market, such as the invention patent with publication number CN111507985A, which discloses an image instance segmentation optimization processing method, device and computer equipment. The method includes: obtaining an image instance segmentation task, the image instance segmentation task carries the image to be segmented and the model identifier; parsing the instance segmentation model corresponding to the model identifier to obtain the original model network corresponding to the instance segmentation model; calling a preset subnetwork, replacing the original model network according to the preset subnetwork, and obtaining a target model network; pruning the preset subnetwork in the target model network to obtain an instance segmentation model after pruning; inputting the image to be segmented into the instance segmentation model after pruning, performing a prediction operation on the image to be segmented, and obtaining an image instance segmentation result.

[0004] Obviously, although the above method can reduce unnecessary computing time consumption, it still has drawbacks. The existing instance segmentation methods are the same, often only using RGB data, and another example is the classic model Mask R-CNN proposed in 2017. However, these methods all have the problem of insufficient instance segmentation accuracy.

[0005] Therefore, it is very necessary to invent an instance segmentation method that can fully utilize depth information. Summary of the invention

[0006] Based on this, it is necessary to provide an instance segmentation method and device based on RGBD images that can achieve accurate instance segmentation in response to the above technical problems.

[0007] The technical solution of the present invention is as follows:

[0008] An instance segmentation method based on RGBD images, the method comprising:

[0009] Step S100: obtaining a current RGB image of a current specific scene, and obtaining actual depth information corresponding to the current RGB image according to the current RGB image;

[0010] Step S200: transmitting the acquired current RGB image and the actual depth information to a workstation having a graphics processing unit (GPU);

[0011] Step S300: The workstation inputs the acquired current RGB image and the actual depth information into two depth residual branches based on convolutional neural networks, and outputs feature maps of four different stages after performing convolution operation on the residual network based on convolutional neural networks; wherein the two depth residual branches are an RGB branch and a depth branch, the current RGB image is input into the RGB branch, and the current RGB image is input into the depth branch; the feature maps of the four different stages are named res2, res3, res4 and res5 according to the network from shallow to deep;

[0012] Step S400: input the feature information res3, res4 and res5 of the RGB branch into the feature pyramid network and obtain new feature maps P3, P4 and P5, and perform specific sampling processing on the feature map P5 to generate feature maps P6 and P7;

[0013] Step S500: sending the feature maps P3 to P7 to the classification branch and the dynamic convolution kernel branch to obtain the category probability that each point on the feature map belongs to a category center point and a dynamic convolution kernel that can represent the instance;

[0014] Step S600: judging whether the category probability is greater than a preset first reference threshold value according to the category probability, if it is judged to be yes, it is considered that there is an object at the point, and then a dynamic depth coordinate convolution operation is performed on the point, and a segmentation mask of the object corresponding to the point is obtained, wherein when there are multiple objects, the segmentation mask is also multiple;

[0015] Step S700: filtering out duplicate objects from the plurality of segmentation masks by non-maximum suppression, and selecting objects whose category probability is greater than a second reference threshold as the final output result.

[0016] Specifically, step S300: the workstation inputs the acquired current RGB image and the actual depth information into two depth residual branches based on convolutional neural networks, and outputs feature maps of four different stages after performing convolution operations on the residual network based on convolutional neural networks; wherein the two depth residual branches are an RGB branch and a depth branch, the current RGB image is input into the RGB branch, and the current RGB image is input into the depth branch; the feature maps of the four different stages are named res2, res3, res4 and res5 according to the network from shallow to deep; specifically including:

[0017] Step S310: After the workstation inputs the acquired current RGB image and the actual depth information into two depth residual branches based on the convolutional neural network respectively, the depth branch performs convolution operation based on the residual network of the convolutional neural network to independently calculate the image feature information layer by layer;

[0018] Step S320: while the RGB branch is performing convolution operation, the image feature information from the depth branch is fused by element-by-element addition multiple times in different stages;

[0019] Step S330: Output feature maps of four different stages after performing convolution operations on the RGB branch and the depth branch.

[0020] Specifically, the framework structure of the convolutional neural network includes:

[0021] a1, the first convolution layer Conv1 of the RGB branch and the first convolution layer Conv1_d of the depth branch perform convolution operations with 64 convolution kernels on the input RGB image and depth image respectively, that is, the outputs of Conv1 and Conv1_d are feature maps with 64 channels;

[0022] b1, Pool1 and Pool1_d are maximum pooling operations with a step size of 2;

[0023] c1, Stage2 and Stage2_d are the output results of Pool1 and Pool1_d after three residual modules without downsampling;

[0024] d1, Stage3 and Stage3_d are the output results of Stage2 and Stage2_d after one residual module operation with downsampling and three residual modules operation without downsampling;

[0025] e1, Stage4 and Stage4_d are the output results of Stage3 and Stage3_d, which are processed by one residual module with downsampling and five residual modules without downsampling;

[0026] f1, Stage5 and Stage5_d are the output results of Stage4 and Stage4_d after one residual module operation with downsampling and two residual modules operation without downsampling;

[0027] Step S320: while the RGB branch is performing convolution operation, the image feature information from the depth branch is fused by element-by-element addition in multiple stages; specifically, the steps include:

[0028] Step S321: perform element-by-element addition operation on the operation result of Pool1_d in the depth branch and the operation result of Pool1 in the RGB branch, and use the added result as the input of Stage2 in the RGB branch;

[0029] Step S322: perform element-by-element addition operation on the calculation result of Stage2_d in the depth branch and the calculation result of Stage2 in the RGB branch, and use the result of the addition as the input of Stage3 in the RGB branch;

[0030] Step S323: perform element-by-element addition operation on the calculation result of Stage3_d in the depth branch and the calculation result of Stage3 in the RGB branch, and use the result of the addition as the input of Stage4 in the RGB branch;

[0031] Step S324: perform an element-by-element addition operation on the calculation result of Stage4_d in the depth branch and the calculation result of Stage4 in the RGB branch, and use the added result as the input of Stage5 in the RGB branch;

[0032] Step S325: perform an element-by-element addition operation on the calculation result of Stage5_d in the depth branch and the calculation result of Stage5 in the RGB branch.

[0033] Specifically, step S500: sending the feature map P3 to the feature map P7 to the classification branch and the dynamic convolution kernel branch to obtain the category probability that each point on the feature map belongs to a category center point and a dynamic convolution kernel that can represent the instance; specifically including:

[0034] Step S510: sending the feature maps P3 to P7 to the classification branch and the dynamic convolution kernel branch, and calculating through the category branch and the dynamic convolution kernel branch in turn, wherein the result of the category branch is activated by the Sigmoid activation function, that is, normalized to between 0 and 1, and the numerical value represents the probability that each point is the center point of an object of a certain category;

[0035] Step S520: The result of the dynamic convolution kernel branch is not processed by the activation function and is directly used as the parameter of the dynamic depth coordinate convolution kernel in step S600.

[0036] Specifically, if the judgment in step S600 is yes, it is considered that there is an object at the point, and then a dynamic depth coordinate convolution operation is performed on the point, and a segmentation mask of the object corresponding to the point is obtained, wherein when there are multiple objects, the segmentation mask is also multiple; specifically including:

[0037] Step S610: performing a 1x1 convolution operation with a step size of 1 and an output channel number of 1 on the feature map P3 after splicing the relative coordinates and the relative depth and the dynamic convolution kernel parameters obtained by the dynamic convolution kernel branch in step S500, and generating an output result;

[0038] Step S620: Activate the output result through the Sigmoid activation function, that is, normalize it to between 0 and 1. This value represents the probability that each point belongs to the object. The output result is a mask probability map.

[0039] Step S630: Taking the probability threshold as 0.5, that is, assigning a value greater than 0.5 in the mask probability map to 1, and vice versa, assigning a value of 0 to obtain a binary image with a value of 0 or 1 as a segmentation mask of the object. When there are multiple objects, the segmentation masks are also multiple.

[0040] Specifically, an instance segmentation device based on an RGBD image comprises:

[0041] The RGB image acquisition module is used to acquire a current RGB image of a current specific scene, and acquire actual depth information corresponding to the current RGB image according to the current RGB image;

[0042] An information transmission module, used for transmitting the acquired current RGB image and the actual depth information to a workstation having a graphics processing unit GPU;

[0043] A feature map output module, used for the workstation to input the acquired current RGB image and the actual depth information into two depth residual branches based on convolutional neural networks, and output feature maps of four different stages after performing convolution operations on the residual network based on convolutional neural networks; wherein the two depth residual branches are an RGB branch and a depth branch, the current RGB image is input into the RGB branch, and the current RGB image is input into the depth branch; the feature maps of the four different stages are named res2, res3, res4 and res5 according to the network from shallow to deep;

[0044] The feature map acquisition module is used to input the feature information res3, res4 and res5 of the RGB branch into the feature pyramid network and obtain new feature maps P3, P4 and P5, and generate feature maps P6 and P7 after performing specific sampling processing on the feature map P5;

[0045] A classification input module is used to input the feature map P3 to the feature map P7 into the classification branch and the dynamic convolution kernel branch to obtain the category probability that each point on the feature map belongs to a category center point and a dynamic convolution kernel that can represent the instance;

[0046] A category judgment module, used to judge whether the category probability is greater than a preset first reference threshold according to the category probability. If it is judged to be yes, it is considered that there is an object at the point, and then a dynamic depth coordinate convolution operation is performed on the point, and a segmentation mask of the object corresponding to the point is obtained, wherein when there are multiple objects, the segmentation mask is also multiple;

[0047] The output result module is used to filter out repeated objects from the multiple segmentation masks through non-maximum suppression, and select the ones with a category probability greater than a second reference threshold as the final output result.

[0048] Specifically, the feature map output module includes:

[0049] An independent computing module, used for the workstation to input the acquired current RGB image and the actual depth information into two depth residual branches based on convolutional neural networks respectively, and the depth branches perform convolution operations based on the residual network of the convolutional neural network to independently calculate layer-by-layer image feature information;

[0050] A feature information fusion module, used for fusing the image feature information from the depth branch by element-by-element addition multiple times in different stages while the RGB branch is performing convolution operation;

[0051] The convolution operation module is used to output feature maps of four different stages after performing convolution operations on the RGB branch and the depth branch.

[0052] Specifically, the feature information fusion module includes:

[0053] An image feature module is used to fuse the image feature information from the depth branch by element-by-element addition multiple times in different stages while the RGB branch is performing convolution operations; specifically, it includes:

[0054] The first operation module is used to perform element-by-element addition operation on the operation result of Pool1_d in the depth branch and the operation result of Pool1 in the RGB branch, and the result of the addition is used as the input of Stage2 in the RGB branch;

[0055] The second operation module is used to perform element-by-element addition operation on the operation result of Stage2_d in the depth branch and the operation result of Stage2 in the RGB branch, and the result of the addition is used as the input of Stage3 in the RGB branch;

[0056] The third operation module is used to perform element-by-element addition operation on the operation result of Stage3_d in the depth branch and the operation result of Stage3 in the RGB branch, and the result of the addition is used as the input of Stage4 in the RGB branch;

[0057] The fourth operation module is used to perform an element-by-element addition operation on the operation result of Stage4_d in the depth branch and the operation result of Stage4 in the RGB branch, and the result of the addition is used as the input of Stage5 in the RGB branch;

[0058] The fifth operation module is used to perform an element-by-element addition operation on the operation result of Stage5_d in the depth branch and the operation result of Stage5 in the RGB branch.

[0059] The present invention also discloses a detection system, comprising a segmentation unit and a detection unit, wherein:

[0060] The segmentation unit is the instance segmentation device based on the RGBD image, which is used to segment the image of the detection target;

[0061] The detection unit is used to detect the image segmented by the segmentation unit.

[0062] The present invention also discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0063] The technical effects achieved by the present invention are as follows:

[0064] 1. The above-mentioned instance segmentation method and device based on RGBD image sequentially obtains the current RGB image of the current specific scene, and obtains the actual depth information corresponding to the current RGB image according to the current RGB image; transmits the obtained current RGB image and the actual depth information to a workstation with a graphics processing unit GPU; the workstation inputs the obtained current RGB image and the actual depth information into two depth residual branches based on convolutional neural network respectively, and outputs feature maps of four different stages after convolution operation on the residual network based on convolutional neural network; wherein the two depth residual branches are RGB branch and depth branch, the current RGB image is input into the RGB branch, and the current RGB image is input into the depth branch; the feature maps of the four different stages are named res2, res3, res4 and res5 according to the network from shallow to deep; the feature information r of the RGB branch is input into the depth branch ... es3, res4 and res5 are input into the feature pyramid network to obtain new feature maps P3, P4 and P5, and feature map P5 is subjected to specific sampling processing to generate feature map P6 and P7; the feature maps P3 to P7 are sent to the classification branch and the dynamic convolution kernel branch to obtain the category probability that each point on the feature map belongs to a category center point and a dynamic convolution kernel that can represent the instance; according to the category probability, it is judged whether the category probability is greater than a preset first reference threshold. If it is judged to be yes, it is considered that there is an object at the point, and then a dynamic depth coordinate convolution operation is performed on the point, and a segmentation mask of the object corresponding to the point is obtained, wherein, when there are multiple objects, the segmentation masks are also multiple; the multiple segmentation masks are filtered out through non-maximum suppression to remove duplicate objects, and the category probability greater than the second reference threshold is selected as the final output result, thereby achieving accurate instance segmentation.

[0065] 2. Innovatively use depth information twice: first, as auxiliary information of RGB in the feature extraction stage and gradually merged with RGB information; second, in the segmentation mask generation stage, the spatial position information contained in the feature map is enhanced through dynamic depth coordinate convolution. Therefore, compared with the traditional RGB-based segmentation model, more accurate instance segmentation can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Schematic diagram of a flow chart of an instance segmentation method based on RGBD images in one embodiment;

[0067] Figure 2 It is a framework structure diagram of the RGB branch and the depth branch in a convolutional neural network in one embodiment;

[0068] Figure 3 The structure of a classification branch and a dynamic convolution kernel branch in one embodiment;

[0069] Figure 4 is a structural block diagram of an instance segmentation device based on RGBD images in one embodiment;

[0070] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0072] In one embodiment, Figure 1 As shown, a method for instance segmentation based on RGBD images is provided, the method comprising:

[0073] Step S100: obtaining a current RGB image of a current specific scene, and obtaining actual depth information corresponding to the current RGB image according to the current RGB image;

[0074] Specifically, the current RGB image and the actual depth information can be obtained using Microsoft's Azure Kinect kit, where the current RGB image retains the original 0-255 RGB three-channel values, and the value of the depth information is proportionally scaled to 0-255. At this time, the current RGB image and the actual depth information are already within the same value range.

[0075] Step S200: transmitting the acquired current RGB image and the actual depth information to a workstation having a graphics processing unit (GPU);

[0076] Specifically, the workstation having a graphics processing unit GPU is pre-selected.

[0077] Step S300: The workstation inputs the acquired current RGB image and the actual depth information into two depth residual branches based on convolutional neural networks, and outputs feature maps of four different stages after performing convolution operation on the residual network based on convolutional neural networks; wherein the two depth residual branches are an RGB branch and a depth branch, the current RGB image is input into the RGB branch, and the current RGB image is input into the depth branch; the feature maps of the four different stages are named res2, res3, res4 and res5 according to the network from shallow to deep;

[0078] Specifically, the convolutional neural network is created based on the Pytorch deep learning library, and its underlying layer is accelerated by Nvidia's CUDA graphics library. The entire system runs on the Ubuntu operating system.

[0079] In this embodiment, except for the first convolution layer of the RGB branch and the depth branch, both of which have a 7x7 convolution kernel, all other convolution layers have a 3x3 convolution kernel.

[0080] Furthermore, the feature layers Stage2, Stage3, Stage4 and Stage5 in the RGB branch and the corresponding Stage2_d, Stage3_d, Stage4_d and Stage5_d of the depth branch are all composed of multiple residual modules, and the residual modules here include residual modules without downsampling and residual modules with downsampling.

[0081] The residual module without downsampling consists of two 3x3 convolutional layers, with a stride and channel multiple of 1. The results of the two convolutional layers are added element-by-element to the input to form a residual module without downsampling.

[0082] In the residual module with downsampling, it consists of two parts. The first part contains two convolutions. The first convolution layer is a 3x3 convolution layer with a stride of 2 and a channel coefficient of 2, and the second convolution layer is a 3x3 convolution layer with a stride of 1 and a channel coefficient of 1.

[0083] The second part contains only one 1x1 convolution with a stride of 1 and a channel coefficient of 2, which takes the input of the module as input.

[0084] Finally, the output of the first part and the output of the second part are added element by element to form a complete residual module with downsampling.

[0085] Step S400: input the feature information res3, res4 and res5 of the RGB branch into the feature pyramid network and obtain new feature maps P3, P4 and P5, and perform specific sampling processing on the feature map P5 to generate feature maps P6 and P7;

[0086] Specifically, in the operation of inputting into the feature pyramid network, Res3, Res4 and Res5 will first undergo a convolution operation with a convolution kernel with 256 1x1s, that is, the number of output channels of the three is 256, and feature maps T3, T4 and P5 are obtained.

[0087] The feature map of P5 is upsampled to twice and then element-wise added to T4 to obtain feature map P4. The obtained P4 is upsampled to twice and then element-wise added to T3 to obtain feature map P3.

[0088] Finally, the obtained feature map P5 is reduced to 1 / 2 and 1 / 4 of the original size through two 0.5 times downsampling, obtaining feature maps P6 and P7.

[0089] Furthermore, the specific sampling process is: downsampling the feature P5 to half the size and one quarter the size, which are named P6 and P7 respectively.

[0090] Step S500: sending the feature maps P3 to P7 to the classification branch and the dynamic convolution kernel branch to obtain the category probability that each point on the feature map belongs to a category center point and a dynamic convolution kernel that can represent the instance;

[0091] Specifically, the relative horizontal and vertical axis coordinates and relative depth value of this point are firstly concatenated with feature P3;

[0092] Then, the dynamic convolution kernel parameters corresponding to the point in the dynamic convolution kernel branch are taken out and a 1x1 convolution operation is performed with the concatenated feature map containing coordinate information and depth information to obtain the mask probability map of the object. Among them, the 1x1 convolution with an output channel of 1 is equivalent to the inner product of the feature of each point on the feature map and the dynamic convolution kernel.

[0093] Step S600: judging whether the category probability is greater than a preset first reference threshold value according to the category probability, if it is judged to be yes, it is considered that there is an object at the point, and then a dynamic depth coordinate convolution operation is performed on the point, and a segmentation mask of the object corresponding to the point is obtained, wherein when there are multiple objects, the segmentation mask is also multiple;

[0094] Specifically, in this embodiment, the first reference threshold is 0.05. If it is greater than 0.05, it is considered that there is an object at the point, and then a dynamic depth coordinate convolution operation is performed on the point.

[0095] Step S700: filtering out duplicate objects from the plurality of segmentation masks by non-maximum suppression, and selecting objects whose category probability is greater than a second reference threshold as the final output result.

[0096] The present invention first obtains the current RGB image of the current specific scene in sequence, and obtains the actual depth information corresponding to the current RGB image according to the current RGB image; transmits the obtained current RGB image and the actual depth information to a workstation with a graphics processing unit (GPU); the workstation inputs the obtained current RGB image and the actual depth information into two depth residual branches based on a convolutional neural network, and outputs feature maps of four different stages after performing convolution operations on the residual network based on the convolutional neural network; wherein the two depth residual branches are an RGB branch and a depth branch, the current RGB image is input into the RGB branch, and the current RGB image is input into the depth branch; the feature maps of the four different stages are named res2, res3, res4 and res5 according to the network from shallow to deep; the feature information res3 and res4 of the RGB branch are respectively connected to the depth branch; the feature information res4 and res5 of the RGB branch are respectively connected to the depth branch; the feature information res5 and res6 of the RGB branch are respectively connected to the depth branch; the feature information res6 and res7 of the RGB branch are respectively connected to the depth branch; the feature information res7 and res8 of the RGB branch are respectively connected to the depth branch; the feature information res8 and res9 of the RGB branch are respectively connected to the depth branch; the feature information res9 and res1 of the RGB branch are respectively connected to the depth branch; the feature information res1 ... res5 is input into the feature pyramid network and new feature maps P3, P4 and P5 are obtained, and feature map P5 is subjected to specific sampling processing to generate feature map P6 and P7; the feature maps P3 to P7 are sent to the classification branch and the dynamic convolution kernel branch to obtain the category probability that each point on the feature map belongs to a category center point and a dynamic convolution kernel that can represent the instance; according to the category probability, it is judged whether the category probability is greater than a preset first reference threshold. If it is judged to be yes, it is considered that there is an object at the point, and then a dynamic depth coordinate convolution operation is performed on the point, and a segmentation mask of the object corresponding to the point is obtained, wherein, when there are multiple objects, the segmentation masks are also multiple; the multiple segmentation masks are filtered out through non-maximum suppression to remove duplicate objects, and the category probability greater than the second reference threshold is selected as the final output result, thereby achieving accurate instance segmentation.

[0097] The present invention also innovatively uses depth information twice: first, as auxiliary information of RGB in the feature extraction stage, it is gradually integrated with RGB information; second, in the segmentation mask generation stage, dynamic depth coordinate convolution is used to enhance the spatial position information contained in the feature map. Therefore, compared with the traditional RGB-based segmentation model, more accurate instance segmentation can be achieved.

[0098] In one embodiment, step S300: the workstation inputs the acquired current RGB image and the actual depth information into two depth residual branches based on a convolutional neural network, and outputs feature maps of four different stages after performing convolution operation on the residual network based on the convolutional neural network; wherein the two depth residual branches are an RGB branch and a depth branch, the current RGB image is input into the RGB branch, and the current RGB image is input into the depth branch; the feature maps of the four different stages are named res2, res3, res4 and res5 according to the network from shallow to deep; specifically including:

[0099] Step S310: After the workstation inputs the acquired current RGB image and the actual depth information into two depth residual branches based on the convolutional neural network respectively, the depth branch performs convolution operation based on the residual network of the convolutional neural network to independently calculate the image feature information layer by layer;

[0100] Specifically, accurate image processing is achieved by independently calculating image feature information layer by layer.

[0101] Step S320: while the RGB branch is performing convolution operation, the image feature information from the depth branch is fused by element-by-element addition multiple times in different stages;

[0102] Specifically, the image feature information from the depth branch is fused by element-by-element addition to achieve refined processing of the image feature information, thereby improving accurate instance segmentation.

[0103] Step S330: Output feature maps of four different stages after performing convolution operations on the RGB branch and the depth branch.

[0104] In one embodiment, Figure 2 As shown, the framework structure of the convolutional neural network includes:

[0105] a1, the first convolution layer Conv1 of the RGB branch and the first convolution layer Conv1_d of the depth branch perform convolution operations with 64 convolution kernels on the input RGB image and depth image respectively, that is, the outputs of Conv1 and Conv1_d are feature maps with 64 channels;

[0106] b1, Pool1 and Pool1_d are maximum pooling operations with a step size of 2;

[0107] c1, Stage2 and Stage2_d are the output results of Pool1 and Pool1_d after three residual modules without downsampling;

[0108] d1, Stage3 and Stage3_d are the output results of Stage2 and Stage2_d after one residual module operation with downsampling and three residual modules operation without downsampling;

[0109] e1, Stage4 and Stage4_d are the output results of Stage3 and Stage3_d, which are processed by one residual module with downsampling and five residual modules without downsampling;

[0110] f1, Stage5 and Stage5_d are the output results of Stage4 and Stage4_d after one residual module operation with downsampling and two residual modules operation without downsampling;

[0111] Step S320: while the RGB branch is performing convolution operation, the image feature information from the depth branch is fused by element-by-element addition in multiple stages; specifically, the steps include:

[0112] Step S321: perform element-by-element addition operation on the operation result of Pool1_d in the depth branch and the operation result of Pool1 in the RGB branch, and use the added result as the input of Stage2 in the RGB branch;

[0113] Step S322: perform element-by-element addition operation on the calculation result of Stage2_d in the depth branch and the calculation result of Stage2 in the RGB branch, and use the result of the addition as the input of Stage3 in the RGB branch;

[0114] Step S323: perform element-by-element addition operation on the calculation result of Stage3_d in the depth branch and the calculation result of Stage3 in the RGB branch, and use the result of the addition as the input of Stage4 in the RGB branch;

[0115] Step S324: perform an element-by-element addition operation on the calculation result of Stage4_d in the depth branch and the calculation result of Stage4 in the RGB branch, and use the added result as the input of Stage5 in the RGB branch;

[0116] Step S325: perform an element-by-element addition operation on the calculation result of Stage5_d in the depth branch and the calculation result of Stage5 in the RGB branch.

[0117] Specifically, step S321 to step S325 are performed based on the framework structure of the convolutional neural network to implement element-by-element addition operation.

[0118] Furthermore, in step S325: after the operation result of Stage5_d in the depth branch and the operation result of Stage5 in the RGB branch are element-by-element added, the result of the addition is input to the next step.

[0119] In one embodiment, step S500: sending the feature maps P3 to P7 to the classification branch and the dynamic convolution kernel branch to obtain the category probability that each point on the feature map belongs to a category center point and a dynamic convolution kernel that can represent the instance; specifically includes:

[0120] Step S510: sending the feature maps P3 to P7 to the classification branch and the dynamic convolution kernel branch, and calculating through the category branch and the dynamic convolution kernel branch in turn, wherein the result of the category branch is activated by the Sigmoid activation function, that is, normalized to between 0 and 1, and the numerical value represents the probability that each point is the center point of an object of a certain category;

[0121] Specifically, Figure 3 As shown in the figure, the classification branch consists of 4 3x3 convolutional layers with a stride of 1 and 1 convolutional layer with a stride of 1x1. The first four convolutional layers keep the number of channels unchanged, while the last 1x1 convolutional layer outputs C channels, that is, the number of categories.

[0122] The dynamic convolution kernel branch is also composed of four 3x3 convolution layers with a stride of 1 and one convolution layer with a stride of 1x1. The first four convolution layers keep the number of channels unchanged, while the last 1x1 convolution layer outputs K channels, which is the number of parameters required for the dynamic convolution kernel.

[0123] Step S520: The result of the dynamic convolution kernel branch is not processed by the activation function and is directly used as the parameter of the dynamic depth coordinate convolution kernel in step S600.

[0124] In one embodiment, if the judgment in step S600 is yes, it is considered that there is an object at the point, and then a dynamic depth coordinate convolution operation is performed on the point, and a segmentation mask of the object corresponding to the point is obtained, wherein when there are multiple objects, the segmentation mask is also multiple; specifically including:

[0125] Step S610: performing a 1x1 convolution operation with a step size of 1 and an output channel number of 1 on the feature map P3 after splicing the relative coordinates and the relative depth and the dynamic convolution kernel parameters obtained by the dynamic convolution kernel branch in step S500, and generating an output result;

[0126] Specifically, by performing a 1x1 convolution operation with a step size of 1 and an output channel number of 1 on the feature map P3 after splicing the relative coordinates and the relative depth and the dynamic convolution kernel parameters obtained by the dynamic convolution kernel branch in step S500, this step is linked to step S500, thereby improving the consistency of data processing and the efficiency of data processing.

[0127] Step S620: Activate the output result through the Sigmoid activation function, that is, normalize it to between 0 and 1. This value represents the probability that each point belongs to the object. The output result is a mask probability map.

[0128] Step S630: Taking the probability threshold as 0.5, that is, assigning a value greater than 0.5 in the mask probability map to 1, and vice versa, assigning a value of 0 to obtain a binary image with a value of 0 or 1 as a segmentation mask of the object. When there are multiple objects, the segmentation masks are also multiple.

[0129] In one embodiment, Figure 4 As shown, an instance segmentation device based on RGBD image, the device comprising:

[0130] The RGB image acquisition module is used to acquire a current RGB image of a current specific scene, and acquire actual depth information corresponding to the current RGB image according to the current RGB image;

[0131] An information transmission module, used for transmitting the acquired current RGB image and the actual depth information to a workstation having a graphics processing unit GPU;

[0132] A feature map output module, used for the workstation to input the acquired current RGB image and the actual depth information into two depth residual branches based on convolutional neural networks, and output feature maps of four different stages after performing convolution operations on the residual network based on convolutional neural networks; wherein the two depth residual branches are an RGB branch and a depth branch, the current RGB image is input into the RGB branch, and the current RGB image is input into the depth branch; the feature maps of the four different stages are named res2, res3, res4 and res5 according to the network from shallow to deep;

[0133] The feature map acquisition module is used to input the feature information res3, res4 and res5 of the RGB branch into the feature pyramid network and obtain new feature maps P3, P4 and P5, and generate feature maps P6 and P7 after performing specific sampling processing on the feature map P5;

[0134] A classification input module is used to input the feature map P3 to the feature map P7 into the classification branch and the dynamic convolution kernel branch to obtain the category probability that each point on the feature map belongs to a category center point and a dynamic convolution kernel that can represent the instance;

[0135] A category judgment module, used to judge whether the category probability is greater than a preset first reference threshold according to the category probability. If it is judged to be yes, it is considered that there is an object at the point, and then a dynamic depth coordinate convolution operation is performed on the point, and a segmentation mask of the object corresponding to the point is obtained, wherein when there are multiple objects, the segmentation mask is also multiple;

[0136] The output result module is used to filter out repeated objects from the multiple segmentation masks through non-maximum suppression, and select the ones with a category probability greater than a second reference threshold as the final output result.

[0137] In one embodiment, the feature map output module includes:

[0138] An independent computing module, used for the workstation to input the acquired current RGB image and the actual depth information into two depth residual branches based on convolutional neural networks respectively, and the depth branches perform convolution operations based on the residual network of the convolutional neural network to independently calculate layer-by-layer image feature information;

[0139] A feature information fusion module, used for fusing the image feature information from the depth branch by element-by-element addition multiple times in different stages while the RGB branch is performing convolution operation;

[0140] The convolution operation module is used to output feature maps of four different stages after performing convolution operations on the RGB branch and the depth branch.

[0141] In one embodiment, the feature information fusion module includes:

[0142] An image feature module is used to fuse the image feature information from the depth branch by element-by-element addition multiple times in different stages while the RGB branch is performing convolution operations; specifically, it includes:

[0143] The first operation module is used to perform element-by-element addition operation on the operation result of Pool1_d in the depth branch and the operation result of Pool1 in the RGB branch, and the result of the addition is used as the input of Stage2 in the RGB branch;

[0144] The second operation module is used to perform element-by-element addition operation on the operation result of Stage2_d in the depth branch and the operation result of Stage2 in the RGB branch, and the result of the addition is used as the input of Stage3 in the RGB branch;

[0145] The third operation module is used to perform element-by-element addition operation on the operation result of Stage3_d in the depth branch and the operation result of Stage3 in the RGB branch, and the result of the addition is used as the input of Stage4 in the RGB branch;

[0146] The fourth operation module is used to perform an element-by-element addition operation on the operation result of Stage4_d in the depth branch and the operation result of Stage4 in the RGB branch, and the result of the addition is used as the input of Stage5 in the RGB branch;

[0147] The fifth operation module is used to perform an element-by-element addition operation on the operation result of Stage5_d in the depth branch and the operation result of Stage5 in the RGB branch.

[0148] In one embodiment, the classification input module is used to: input the feature map P3 to the feature map P7 into the classification branch and the dynamic convolution kernel branch, and calculate through the category branch and the dynamic convolution kernel branch in turn, wherein the result of the category branch is activated by the Sigmoid activation function, that is, normalized to between 0 and 1, and the numerical value represents the probability that each point is the center point of an object of a certain category; the result of the dynamic convolution kernel branch is not processed by the activation function, and is directly used as the parameter of the dynamic depth coordinate convolution kernel in step S600.

[0149] In one embodiment, the category judgment module is used to: perform a 1x1 convolution operation with a step size of 1 and an output channel number of 1 on the feature map P3 after splicing the relative coordinates and the relative depth and the dynamic convolution kernel parameters obtained by the dynamic convolution kernel branch in step S500, and generate an output result; activate the output result through a Sigmoid activation function, that is, normalize it to between 0-1, and this value represents the probability that each point belongs to the object, and the output result is a mask probability map; take the probability threshold as 0.5, that is, assign a value greater than 0.5 in the mask probability map to 1, otherwise assign a value of 0, and obtain a binary image with a value of 0 or 1 as a segmentation mask of the object, wherein, when there are multiple objects, the segmentation mask is also multiple.

[0150] Based on the above embodiments, the present invention also discloses a detection system, including a segmentation unit and a detection unit, wherein: the segmentation unit is the instance segmentation device based on RGBD image, which is used to segment the image of the detection target; the detection unit is used to detect the image segmented by the segmentation unit.

[0151] In one embodiment, Figure 5 As shown, a computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned instance segmentation method based on RGBD image are implemented.

[0152] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned instance segmentation method based on RGBD images.

[0153] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0154] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An instance segmentation method based on RGBD images, characterized in that: The method comprises: Step S100: obtaining a current RGB image of a current specific scene, and obtaining actual depth information corresponding to the current RGB image according to the current RGB image; Step S200: transmitting the acquired current RGB image and the actual depth information to a workstation having a graphics processing unit (GPU); Step S300: The workstation inputs the acquired current RGB image and the actual depth information into two depth residual branches based on convolutional neural networks, and outputs feature maps of four different stages after performing convolution operation on the residual network based on convolutional neural networks; wherein the two depth residual branches are an RGB branch and a depth branch, the current RGB image is input into the RGB branch, and the current RGB image is input into the depth branch; the feature maps of the four different stages are named res2, res3, res4 and res5 according to the network from shallow to deep; Step S400: input the feature information res3, res4 and res5 of the RGB branch into the feature pyramid network and obtain new feature maps P3, P4 and P5, and perform specific sampling processing on the feature map P5 to generate feature maps P6 and P7; Step S500: sending the feature maps P3 to P7 to the classification branch and the dynamic convolution kernel branch to obtain the category probability that each point on the feature map belongs to a category center point and a dynamic convolution kernel that can represent the instance; Step S600: judging whether the category probability is greater than a preset first reference threshold value according to the category probability, if it is judged to be yes, it is considered that there is an object at the point, and then a dynamic depth coordinate convolution operation is performed on the point, and a segmentation mask of the object corresponding to the point is obtained, wherein when there are multiple objects, the segmentation mask is also multiple; Step S700: filtering out duplicate objects from the plurality of segmentation masks by non-maximum suppression, and selecting objects whose category probability is greater than a second reference threshold as the final output result.

2. The instance segmentation method based on RGBD image according to claim 1, characterized in that: Step S300: The workstation inputs the acquired current RGB image and the actual depth information into two depth residual branches based on convolutional neural networks, and outputs feature maps of four different stages after performing convolution operation on the residual network based on convolutional neural networks; wherein the two depth residual branches are RGB branch and depth branch, the current RGB image is input into the RGB branch, and the current RGB image is input into the depth branch; the feature maps of the four different stages are named res2, res3, res4 and res5 according to the network from shallow to deep; specifically including: Step S310: After the workstation inputs the acquired current RGB image and the actual depth information into two depth residual branches based on the convolutional neural network respectively, the depth branch performs convolution operation based on the residual network of the convolutional neural network to independently calculate the image feature information layer by layer; Step S320: while the RGB branch is performing convolution operation, the image feature information from the depth branch is fused by element-by-element addition multiple times in different stages; Step S330: Output feature maps of four different stages after performing convolution operations on the RGB branch and the depth branch.

3. The instance segmentation method based on RGBD image according to claim 2, characterized in that: The framework structure of the convolutional neural network includes: a1, the first convolution layer Conv1 of the RGB branch and the first convolution layer Conv1_d of the depth branch perform convolution operations with 64 convolution kernels on the input RGB image and depth image respectively, that is, the outputs of Conv1 and Conv1_d are feature maps with 64 channels; b1, Pool1 and Pool1_d are maximum pooling operations with a step size of 2; c1, Stage2 and Stage2_d are the output results of Pool1 and Pool1_d after three residual modules without downsampling; d1, Stage3 and Stage3_d are the output results of Stage2 and Stage2_d after one residual module operation with downsampling and three residual modules operation without downsampling; e1, Stage4 and Stage4_d are the output results of Stage3 and Stage3_d, which are processed by one residual module with downsampling and five residual modules without downsampling; f1, Stage5 and Stage5_d are the output results of Stage4 and Stage4_d after one residual module operation with downsampling and two residual modules operation without downsampling; Step S320: while the RGB branch is performing convolution operation, the image feature information from the depth branch is fused by element-by-element addition in multiple stages; specifically, the steps include: Step S321: perform element-by-element addition operation on the operation result of Pool1_d in the depth branch and the operation result of Pool1 in the RGB branch, and use the added result as the input of Stage2 in the RGB branch; Step S322: perform element-by-element addition operation on the calculation result of Stage2_d in the depth branch and the calculation result of Stage2 in the RGB branch, and use the result of the addition as the input of Stage3 in the RGB branch; Step S323: perform element-by-element addition operation on the calculation result of Stage3_d in the depth branch and the calculation result of Stage3 in the RGB branch, and use the result of the addition as the input of Stage4 in the RGB branch; Step S324: perform an element-by-element addition operation on the calculation result of Stage4_d in the depth branch and the calculation result of Stage4 in the RGB branch, and use the added result as the input of Stage5 in the RGB branch; Step S325: perform an element-by-element addition operation on the calculation result of Stage5_d in the depth branch and the calculation result of Stage5 in the RGB branch.

4. The instance segmentation method based on RGBD image according to claim 1, characterized in that: Step S500: sending the feature maps P3 to P7 to the classification branch and the dynamic convolution kernel branch to obtain the category probability that each point on the feature map belongs to a category center point and a dynamic convolution kernel that can represent the instance; specifically including: Step S510: sending the feature maps P3 to P7 to the classification branch and the dynamic convolution kernel branch, and calculating through the category branch and the dynamic convolution kernel branch in turn, wherein the result of the category branch is activated by the Sigmoid activation function, that is, normalized to between 0 and 1, and the numerical value represents the probability that each point is the center point of an object of a certain category; Step S520: The result of the dynamic convolution kernel branch is not processed by the activation function and is directly used as the parameter of the dynamic depth coordinate convolution kernel in step S600.

5. The instance segmentation method based on RGBD image according to claim 4, characterized in that: If the judgment in step S600 is yes, it is considered that there is an object at the point, and then a dynamic depth coordinate convolution operation is performed on the point, and a segmentation mask of the object corresponding to the point is obtained, wherein when there are multiple objects, the segmentation mask is also multiple; specifically including: Step S610: performing a 1x1 convolution operation with a step size of 1 and an output channel number of 1 on the feature map P3 after splicing the relative coordinates and the relative depth and the dynamic convolution kernel parameters obtained by the dynamic convolution kernel branch in step S500, and generating an output result; Step S620: Activate the output result through the Sigmoid activation function, that is, normalize it to between 0 and 1. This value represents the probability that each point belongs to the object. The output result is a mask probability map. Step S630: Taking the probability threshold as 0.5, that is, assigning a value greater than 0.5 in the mask probability map to 1, and vice versa, assigning a value of 0 to obtain a binary image with a value of 0 or 1 as a segmentation mask of the object. When there are multiple objects, the segmentation masks are also multiple.

6. An instance segmentation device based on RGBD image, characterized in that: The device comprises: The RGB image acquisition module is used to acquire a current RGB image of a current specific scene, and acquire actual depth information corresponding to the current RGB image according to the current RGB image; An information transmission module, used for transmitting the acquired current RGB image and the actual depth information to a workstation having a graphics processing unit GPU; A feature map output module, used for the workstation to input the acquired current RGB image and the actual depth information into two depth residual branches based on convolutional neural networks, and output feature maps of four different stages after performing convolution operations on the residual network based on convolutional neural networks; wherein the two depth residual branches are an RGB branch and a depth branch, the current RGB image is input into the RGB branch, and the current RGB image is input into the depth branch; the feature maps of the four different stages are named res2, res3, res4 and res5 according to the network from shallow to deep; The feature map acquisition module is used to input the feature information res3, res4 and res5 of the RGB branch into the feature pyramid network and obtain new feature maps P3, P4 and P5, and generate feature maps P6 and P7 after performing specific sampling processing on the feature map P5; A classification input module is used to input the feature map P3 to the feature map P7 into the classification branch and the dynamic convolution kernel branch to obtain the category probability that each point on the feature map belongs to a category center point and a dynamic convolution kernel that can represent the instance; A category judgment module, used to judge whether the category probability is greater than a preset first reference threshold according to the category probability. If it is judged to be yes, it is considered that there is an object at the point, and then a dynamic depth coordinate convolution operation is performed on the point, and a segmentation mask of the object corresponding to the point is obtained, wherein when there are multiple objects, the segmentation mask is also multiple; The output result module is used to filter out repeated objects from the multiple segmentation masks through non-maximum suppression, and select the category probability greater than the second reference threshold as the final output result.

7. The RGBD image-based instance segmentation device according to claim 6, characterized in that: The feature map output module comprises: An independent computing module, used for the workstation to input the acquired current RGB image and the actual depth information into two depth residual branches based on convolutional neural networks respectively, and the depth branches perform convolution operations based on the residual network of the convolutional neural network to independently calculate layer-by-layer image feature information; A feature information fusion module, used for fusing the image feature information from the depth branch by element-by-element addition multiple times in different stages while the RGB branch is performing convolution operation; The convolution operation module is used to output feature maps of four different stages after performing convolution operations on the RGB branch and the depth branch.

8. The RGBD image-based instance segmentation device according to claim 7, characterized in that: The feature information fusion module includes: An image feature module is used to fuse the image feature information from the depth branch by element-by-element addition multiple times in different stages while the RGB branch is performing convolution operations; specifically, it includes: The first operation module is used to perform element-by-element addition operation on the operation result of Pool1_d in the depth branch and the operation result of Pool1 in the RGB branch, and the result of the addition is used as the input of Stage2 in the RGB branch; The second operation module is used to perform element-by-element addition operation on the operation result of Stage2_d in the depth branch and the operation result of Stage2 in the RGB branch, and the result of the addition is used as the input of Stage3 in the RGB branch; The third operation module is used to perform element-by-element addition operation on the operation result of Stage3_d in the depth branch and the operation result of Stage3 in the RGB branch, and the result of the addition is used as the input of Stage4 in the RGB branch; The fourth operation module is used to perform an element-by-element addition operation on the operation result of Stage4_d in the depth branch and the operation result of Stage4 in the RGB branch, and the result of the addition is used as the input of Stage5 in the RGB branch; The fifth operation module is used to perform an element-by-element addition operation on the operation result of Stage5_d in the depth branch and the operation result of Stage5 in the RGB branch.

9. A detection system, comprising a segmentation unit and a detection unit, characterized in that: The segmentation unit is an instance segmentation device based on RGBD image according to any one of claims 6 to 8, used for segmenting the image of the detection target; The detection unit is used to detect the image segmented by the segmentation unit.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images