A Panoptic Image Segmentation Method Based on Color and Depth Information

Abstract

The present invention discloses a method for image panoramic segmentation based on color and depth information, including S1, inputting a binocular RGB image to generate a depth map, and performing feature extraction to obtain color features and depth features; S2, fusing the color features and depth features with color and depth information to obtain fused features; S3, processing the fused features to generate a position head and a weight head of the image respectively; S4, using a position weight fusion operation, merging the weight heads, and generating object weights and background weights; S5, processing the fused features to obtain a high-resolution feature map; S6, fusing the object weights and background weights with the high-resolution feature map, and upsampling to obtain a panoramic segmentation result. The present invention combines the two branches of semantic segmentation and instance segmentation in combination with the color and depth information of the image, and regards the background in the image as a continuous object instance, thereby achieving consistency between semantic segmentation and instance segmentation, avoiding complex post-processing, and realizing end-to-end training and prediction.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image panorama segmentation method based on color and depth information. Background Art

[0002] In the field of computer vision, the task of image panoptic segmentation is to assign a semantic label and an instance label to each pixel in the image. Panoptic segmentation can not only identify the type of each object (thing), but also distinguish different instances of the same type. At the same time, panoptic segmentation can also identify the background (stuff) in the image. Therefore, panoptic segmentation is a combination of semantic segmentation and instance segmentation, and has a wide range of applications in autonomous driving, video surveillance, medical image analysis, remote sensing image analysis and other fields.

[0003] Existing panoramic segmentation methods usually use CNN (such as ResNet-50, FPN, etc.) as the backbone network to extract image features, and input them into the semantic segmentation (such as FPN) and instance segmentation (such as Mask R-CNN) branches to obtain image semantic segmentation and instance segmentation results respectively. Finally, the two branch results are merged to obtain the panoramic segmentation result.

[0004] There are two main problems with existing panoramic segmentation methods: First, semantic segmentation and instance segmentation are considered as two independent tasks, and the outputs of the two tasks are merged and conflict-free using post-processing methods, which makes the entire system not an end-to-end structure. Second, existing methods mainly process RGB images. When two similar objects are close together in an image, they are often mistaken for one object. Therefore, a method for image panoramic segmentation based on color and depth information is proposed. Summary of the invention

[0005] The object of the present invention is to provide a method for panoramic image segmentation based on color and depth information to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for image panorama segmentation based on color and depth information, the segmentation method comprising the following steps:

[0007] S1, input binocular RGB image, use stereo matching algorithm to generate corresponding depth map, and perform feature extraction to obtain color feature P corresponding to the image i c and deep features P i d ;

[0008] The operation process of S1 includes:

[0009] S101. First, read in a binocular RGB image of the same scene captured by a stereo camera, expressed as:

[0010] I c =(I c1 , I c2 ) (1);

[0011] S102, using a stereo matching algorithm to process the read RGB image to obtain a depth image I corresponding to the image d ;

[0012] S103, use the feature pyramid network with Resnet50 as the backbone network to extract features of multiple scales of the image, denoted as P i , represents the characteristics of the image at scale i, its length and width are 1 / 2 of the original image i , and the extracted features include P2~P5;

[0013] S104, performing two more convolution operations on P5 to obtain image features P6 and P7;

[0014] S105, I c and I d Through the above feature pyramid network, we get:

[0015] P c = {P i c},i∈{2,3,…,7} (2);

[0016] P d = {P i d},i∈{2,3,…,7} (3);

[0017] Among them, P i c For I c The embedded features of each scale obtained by the feature pyramid network, P i d For I d Features of each scale obtained through the feature pyramid network;

[0018] S2, fusing the color feature and depth feature obtained in S1 to obtain a fused feature;

[0019] The operation process of S2 includes:

[0020] S201, first, at each scale, the feature P i c and P i dUse algorithmic averaging for fusion;

[0021] S202, then, the fused features are convolved to obtain embedded features that fuse color and depth information:

[0022] X={X i},i∈{2,3,…,7} (4);

[0023] Its mathematical expression is:

[0024] X i =conv 1×1 (0.5*(P i c +P i d )) (5);

[0025] S3, processes the fusion features obtained in S2 to generate the position head G of the image respectively t and G s and weight head W i ;

[0026] S4, using the position weight fusion operation, merges the repeated weight heads of each scale obtained in S3 to generate object weights and background weights;

[0027] S5, the fusion features obtained in S2 are processed again to obtain a high-resolution feature map X hf ;

[0028] S6, fuses the object weight and background weight obtained in S4 with the high-resolution feature map obtained in S5, performs an upsampling operation, and obtains a panoramic segmentation result.

[0029] Preferably, the processing step S3 first extracts a subset from the fused embedded features, which is expressed as:

[0030] X′={X i},i∈{4,5,6,7} (6)

[0031] Then, a position head and a weight head are generated respectively, where the position head is used to locate and classify the objects and background in the image, and the weight head is used to predict the weight of each spatial position in X′.

[0032] Preferably, the method for generating the location header includes:

[0033] A1, for each scale feature map X i , perform convolution operation;

[0034] A2, after each convolution operation, uses group normalization operation and ReLU activation function, and is divided into two branches;

[0035] A3, using convolution operation to generate N t object position heat map, each heat map represents X i The probability that each pixel belongs to the center of an object instance is given by specifying the probability threshold and using the local non-maximum constraint method. The heat map is converted into a binary image, expressed as:

[0036]

[0037] in, Represents the feature map X i The corresponding binary image of the center of an instance of the j-th object;

[0038] A4, the other way uses convolution operation to generate N s Background position heatmaps, each heatmap represents X i The probability that each pixel belongs to a certain type of background. After specifying the probability threshold, the heat map is converted into a binary image, which is expressed as:

[0039]

[0040] in, Represents the feature map X i The binary image corresponding to the j-th background.

[0041] Preferably, the method for generating the weighted head includes:

[0042] B1, for each scale feature map X i , add the learnable relative position information, and use a feature map to represent the x-axis and y-axis coordinates respectively. i Aggregate the channel dimension to obtain the embedded feature containing position information, denoted as Y i ;

[0043] B2, for Y i Perform convolution operation to obtain the weight head W i .

[0044] Preferably, the processing process of S4 includes:

[0045] S401, for background class k and feature map X i ,Will Pixels with a median value of 1 correspond to W i The values ​​at the corresponding positions of are averaged to obtain the background weight, which is calculated as follows:

[0046]

[0047] Among them, avg() represents the average calculation operation; multiply() represents the matrix multiplication operation;

[0048] S402, perform mean pooling operation on the results of feature maps of each scale to obtain N s Background weight for background-like

[0049] S403, for the object instance k and feature map X i , take W i Corresponding The value of the pixel position with a value of 1 is used to obtain the object weight, which is calculated as follows:

[0050]

[0051] Among them, (x, y) satisfies:

[0052]

[0053] S404, calculated at different scales If the two weight heads W i and W j If the cosine similarity between them is higher than the specified threshold, the two object instances are merged by the mean, and finally the top M object instances are selected according to their confidence scores, and their corresponding object weights are calculated.

[0054] Preferably, the processing of S5 includes:

[0055] S501, first, extract a subset from the fused embedding features:

[0056] X″={X i},i∈{2,3,4,5} (12);

[0057] S502, then, to X i Use (5-i) upsampling operations, each of which includes three layers of convolution, group normalization, and ReLU activation function, and convert the features of each layer to the same size;

[0058] S503, performing positional addition on the converted features;

[0059] S504, the added features are subjected to three-layer convolution operations to obtain a high-resolution feature map X hf .

[0060] Preferably, the processing of S6 includes:

[0061] S601, for the high-resolution feature map X hf The same method is used to generate the weight head, adding the learnable relative position information and performing convolution operation to obtain X hf′ ;

[0062] S602, for N s Background weight and M object weights Respectively with X hf′ Perform convolution operation to get N s +M prediction images, perform bilinear difference upsampling and threshold operation on the prediction images to obtain binary prediction images of the same size as the original images;

[0063] S603, processing the conflicting pixels in each prediction image, and marking the pixels to which no type is assigned as background class.

[0064] Preferably, the N obtained in S602 s +M prediction images, each of which corresponds to an instance or a background in the image.

[0065] Technical effects and advantages of the present invention:

[0066] (1) The present invention combines the semantic segmentation and instance segmentation branches by combining the color and depth information of the image, and regards the background in the image as a continuous object instance, thereby achieving consistency between semantic segmentation and instance segmentation, avoiding complex post-processing, and realizing end-to-end training and prediction;

[0067] (2) The present invention introduces additional image depth information, which can correctly distinguish different instances of the same object by combining the color and depth information of the object in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 4 is an operational flow chart of the segmentation method of the present invention.

[0069] Figure 2 FIG. 4 is an operational architecture diagram of the segmentation method of the present invention.

[0070] Figure 3 Generate a schematic diagram for the inventive location head.

[0071] Figure 4 Generate a schematic diagram for the weighted head of the present invention.

[0072] Figure 5 Generate schematics for the high resolution features of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] The present invention provides Figure 1-2 The method for panoramic image segmentation based on color and depth information shown in FIG. 1 includes the following steps:

[0075] S1, input binocular RGB image, use stereo matching algorithm to generate corresponding depth map, and perform feature extraction to obtain color feature P corresponding to the image i c and deep features P i d ;

[0076] Specifically, the operation process of S includes:

[0077] S101. First, read in a binocular RGB image of the same scene captured by a stereo camera, expressed as:

[0078] I c =(I c1 , I c2 ) (1);

[0079] It should be noted that a stereo camera is a camera system with dual cameras or dual lenses for capturing stereoscopic images or videos. It works similarly to the human binocular vision system, obtaining depth information by simultaneously photographing the same scene from two different positions or angles. Benefits and functions include: Stereo and depth information. Stereo cameras can capture the distance and depth information of objects in the scene, which is very important for identifying and analyzing the position and shape of objects. For example, in computer vision and augmented reality, this depth information is very useful for object tracking, scene reconstruction, and virtual reality. 3D reconstruction and measurement. By analyzing images captured from different perspectives, a 3D model of the scene can be reconstructed and accurate size and distance measurements can be made. This has a wide range of applications in industrial design, architecture, archeology, and medical imaging. Depth of field control. Through the dual-camera setting, more refined control of depth of field can be achieved. This is very helpful for shooting artworks or professional photography that require a specific depth of field effect. Environmental perception and navigation. In self-driving cars and robotics, stereo cameras can help vehicles or robots identify and avoid obstacles, and achieve accurate navigation and environmental perception.

[0080] S102, using a stereo matching algorithm to process the read RGB image to obtain a depth image I corresponding to the image d ;

[0081] It should be noted that the stereo matching algorithm (SGBM) is a matching algorithm for stereo vision, which is often used to process images taken by stereo cameras. Its manifestations include: disparity map. The SGBM algorithm calculates disparity by comparing the pixels of the left and right images, that is, the pixel offset of the same object in the two images. These disparity values ​​can be used to generate disparity maps to display the depth information of the object. Parameter adjustment. There are some parameters in the SGBM algorithm that can be adjusted, such as disparity range, window size, etc., which can be adjusted according to actual conditions to obtain better matching results. Its benefits and functions include: depth map generation. The disparity map calculated by the SGBM algorithm can be further converted into a depth map to display the distance information of the object in the scene, which is very useful for three-dimensional reconstruction and environmental perception. Object recognition and segmentation. Based on the depth information generated by the SGBM algorithm, it can help identify and segment different objects in the scene, which is of great significance to computer vision, autonomous driving and other fields. Stereo matching accuracy. The SGBM algorithm has high accuracy and stability when processing stereo image matching, and can effectively handle complex scenes and areas with weak textures. Real-time performance, the SGBM algorithm can achieve real-time processing with hardware acceleration, and is suitable for application scenarios that require fast feedback, such as real-time monitoring, virtual reality, etc.

[0082] S103, use the feature pyramid network with Resnet50 as the backbone network to extract features of multiple scales of the image, denoted as P i , represents the characteristics of the image at scale i, its length and width are 1 / 2 of the original image i , and the extracted features include P2~P5;

[0083] S104, perform two more convolution operations on P5 (the convolution kernel size is 3×3 and the step size is 2) to obtain image features P6 and P7;

[0084] S105, I c and I d Through the above feature pyramid network, we get:

[0085] P c = {P i c},i∈{2,3,…,7} (2);

[0086] P d = {P i d},i∈{2,3,…,7} (3);

[0087] Among them, P i c For I c The embedded features of each scale obtained by the feature pyramid network, P i d For I d Features of each scale obtained through the feature pyramid network.

[0088] S2, fusing the color feature and depth feature obtained in S1 to obtain a fused feature;

[0089] Specifically, the operation process of S2 includes:

[0090] S201, first, at each scale, the feature P i c and P i d Use algorithmic averaging for fusion;

[0091] S202, then, the fused features are convolved (the convolution kernel size is 1×1 and the step size is 1) to obtain the embedded features of the fused color and depth information:

[0092] X={X i},i∈{2,3,…,7} (4);

[0093] Its mathematical expression is:

[0094] X i =conv 1×1 (0.5*(P i c +P i d )) (5).

[0095] S3, processes the fusion features obtained in S2 to generate the position head G of the image respectively t and G s and weight head W i ;

[0096] Specifically, the processing of S3 first extracts a subset from the fused embedding features, expressed as:

[0097] X′={X i},i∈{4,5,6,7} (6)

[0098] Then, a position head and a weight head are generated respectively, where the position head is used to locate and classify the objects and background in the image, and the weight head is used to predict the weight of each spatial position in X′.

[0099] For further reference, Figure 3 As shown, the method for generating the location header includes:

[0100] A1, for each scale feature map X i , perform convolution operation (including 3 convolution layers, the convolution kernel size is 3×3 and the step size is 1);

[0101] A2, in order to reduce the impact of batch size on the model and the problem of internal covariate shift in the network, group normalization and ReLU activation function are used after each convolution operation, and it is divided into two branches;

[0102] It should be noted that the calculation of the Relu function is performed after convolution, so it is a nonlinear activation function like the tanh function and the sigmoid function. The reciprocal of the Relu function is always equal to 1 in the positive part, so using the Relu activation function in a deep network will not cause the problem of small gradients and explosions. In addition, the Relu function has a fast calculation speed, which speeds up the training of the network. Its expression is:

[0103]

[0104] A3, all the way using convolution operation (convolution kernel size is 3×3, step size is 1, number of output channels is N t 、N t Generate N for the number of object categories) t object position heat map, each heat map represents X i The probability that each pixel belongs to the center of an object instance is given by specifying the probability threshold and using the local non-maximum constraint method. The heat map is converted into a binary image, expressed as:

[0105]

[0106] in, Represents the feature map X i The corresponding binary image of the center of an instance of the j-th object;

[0107] A4, the other path uses convolution operation (convolution kernel size is 3×3, step size is 1, number of output channels is N s 、N s is the number of background categories) s Background position heatmaps, each heatmap represents X i The probability that each pixel belongs to a certain type of background. After specifying the probability threshold, the heat map is converted into a binary image, which is expressed as:

[0108]

[0109] in, Represents the feature map X i The binary image corresponding to the j-th background.

[0110] For further reference, Figure 4 As shown, the method for generating the weight head includes:

[0111] B1, for each scale feature map X i , add the learnable relative position information, and use a feature map to represent the x-axis and y-axis coordinates respectively. i Aggregate the channel dimension to obtain the embedded feature containing position information, denoted as Y i ;

[0112] B2, for Y i Perform a convolution operation (including 3 convolution layers, the convolution kernel size is 3×3, and the step size is 1) to obtain the weight head W i .

[0113] S4, using the position weight fusion operation, merges the repeated weight heads of each scale obtained in S3 to generate object weights and background weights;

[0114] Specifically, the processing of S4 includes:

[0115] S401, for background class k and feature map X i ,Will Pixels with a median value of 1 correspond to W i The values ​​at the corresponding positions of are averaged to obtain the background weight, which is calculated as follows:

[0116]

[0117] Among them, avg() represents the average calculation operation; multiply() represents the matrix multiplication operation;

[0118] S402, perform mean pooling operation on the results of feature maps of each scale to obtain N s Background weight for background-like

[0119] S403, for the object instance k and feature map X i , take W i Corresponding The value of the pixel position with a value of 1 is used to obtain the object weight, which is calculated as follows:

[0120]

[0121] Among them, (x, y) satisfies:

[0122]

[0123] S404, calculated at different scales If the two weight heads W i and W j If the cosine similarity between them is higher than the specified threshold, the two object instances are merged by the mean, and finally the top M object instances are selected according to their confidence scores, and their corresponding object weights are calculated.

[0124] S5, the fusion features obtained in S2 are processed again to obtain a high-resolution feature map X hf ;

[0125] Specifically, refer to Figure 5 As shown, the processing of S5 includes:

[0126] S501, first, extract a subset from the fused embedding features:

[0127] X″={X i},i∈{2,3,4,5} (13);

[0128] S502, then, to X i Use (5-i) upsampling operations, each of which includes three layers of convolution (convolution kernel size is 3×3, stride is 1), group normalization and ReLU activation function, and convert the features of each layer to the same size;

[0129] S503, performing positional addition on the converted features;

[0130] S504, the added features are subjected to three-layer convolution operation (the convolution kernel size is 3×3 and the step size is 1) to obtain a high-resolution feature map X hf .

[0131] S6, fuses the object weight and background weight obtained in S4 with the high-resolution feature map obtained in S5, performs an upsampling operation, and obtains a panoramic segmentation result.

[0132] Specifically, the processing of S6 includes:

[0133] S601, for the high-resolution feature map X hf The same method is used to generate the weight head, adding the learnable relative position information and performing convolution operation to obtain X hf′ ;

[0134] S602, for N s Background weight and M object weights Respectively with Xhf′ Perform convolution operation to get N s +M prediction images, perform bilinear difference upsampling and threshold operation on the prediction images to obtain binary prediction images of the same size as the original images;

[0135] Furthermore, N obtained in S602 s +M prediction images, each of which corresponds to an instance or a background in the image.

[0136] S603, processing the conflicting pixels in each prediction image, and marking the pixels to which no type is assigned as background class.

[0137] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for image panoptic segmentation based on color and depth information, characterized in that: The segmentation method comprises the following steps: S1, input binocular RGB image, use stereo matching algorithm to generate corresponding depth map, and perform feature extraction to obtain color feature P corresponding to the image i c and deep features P i d ; The operation process of S1 includes: S101. First, read in a binocular RGB image of the same scene captured by a stereo camera, expressed as: I c =(I c1 ,I c2 ) (1); S102, using a stereo matching algorithm to process the read RGB image to obtain a depth image I corresponding to the image d ; S103, use the feature pyramid network with Resnet50 as the backbone network to extract features of multiple scales of the image, denoted as P i , represents the characteristics of the image at scale i, its length and width are 1 / 2 of the original image i , and the extracted features include P2~P5; S104, performing two more convolution operations on P5 to obtain image features P6 and P7; S105, I c and I d Through the above feature pyramid network, we get: P c ={P i c },i∈{2,3,…,7} (2); P d ={P i d },i∈{2,3,…,7} (3); Among them, P i c For I c The embedded features of each scale obtained by the feature pyramid network, P i d For I d Features of each scale obtained through the feature pyramid network; S2, fusing the color feature and depth feature obtained in S1 to obtain a fused feature; The operation process of S2 includes: S201, first, at each scale, the feature P i c and P i d Use algorithmic averaging for fusion; S202, then, the fused features are convolved to obtain embedded features that fuse color and depth information: X={X i },i∈{2,3,…,7} (4); Its mathematical expression is: X i =conv 1×1 (0.5*(P i c +P i d )) (5); S3, processes the fusion features obtained in S2 to generate the position head G of the image respectively t and G s and weight head W i ; S4, using the position weight fusion operation, merges the repeated weight heads of each scale obtained in S3 to generate object weights and background weights; S5, the fusion features obtained in S2 are processed again to obtain a high-resolution feature map X hf ; S6, fuses the object weight and background weight obtained in S4 with the high-resolution feature map obtained in S5, performs an upsampling operation, and obtains a panoramic segmentation result.

2. The method for panoramic image segmentation based on color and depth information according to claim 1, characterized in that: The processing of S3 first extracts a subset from the fused embedded features, expressed as: X′={X i },i∈{4,5,6,7} (6) Then, a position head and a weight head are generated respectively, where the position head is used to locate and classify the objects and background in the image, and the weight head is used to predict the weight of each spatial position in X′.

3. The method for image panoptic segmentation based on color and depth information according to claim 2, characterized in that: The method for generating the location header includes: A1, for each scale feature map X i , perform convolution operation; A2, after each convolution operation, uses group normalization operation and ReLU activation function, and is divided into two branches; A3, using convolution operation to generate N t object position heat map, each heat map represents X i The probability that each pixel belongs to the center of an object instance is given by specifying the probability threshold and using the local non-maximum constraint method. The heat map is converted into a binary image, expressed as: in, Represents the feature map X i The corresponding binary image of the center of an instance of the j-th object; A4, the other way uses convolution operation to generate N s Background position heatmaps, each heatmap represents X i The probability that each pixel belongs to a certain type of background. After specifying the probability threshold, the heat map is converted into a binary image, which is expressed as: in, Represents the feature map X i The binary image corresponding to the j-th background.

4. The method for image panoptic segmentation based on color and depth information according to claim 2, characterized in that: The method for generating the weighted head includes: B1, for each scale feature map X i , add the learnable relative position information, and use a feature map to represent the x-axis and y-axis coordinates respectively. i Aggregate the channel dimension to obtain the embedded feature containing position information, denoted as Y i ; B2, for Y i Perform convolution operation to get the weight head W i .

5. The method for panoramic image segmentation based on color and depth information according to claim 3, characterized in that: The processing of S4 includes: S401, for background class k and feature map X i ,Will Pixels with a median value of 1 correspond to W i The values ​​at the corresponding positions of are averaged to obtain the background weight, which is calculated as follows: Among them, avg() represents the average calculation operation; multiply() represents the matrix multiplication operation; S402, perform mean pooling operation on the results of feature maps of each scale to obtain N s Background weight for background-like S403, for the object instance k and feature map X i , take W i Corresponding The value of the pixel position with a value of 1 is used to obtain the object weight, which is calculated as follows: Among them, (x, y) satisfies: S404, calculated at different scales If the two weight heads W i and W j If the cosine similarity between them is higher than the specified threshold, the two object instances are merged by the mean, and finally the top M object instances are selected according to their confidence scores, and their corresponding object weights are calculated.

6. The method for panoramic image segmentation based on color and depth information according to claim 3, characterized in that: The processing of S5 includes: S501, first, extract a subset from the fused embedding features: X″={X i },i∈{2,3,4,5} (12); S502, then, to X i Use (5-i) upsampling operations, each of which includes three layers of convolution, group normalization, and ReLU activation function, and convert the features of each layer to the same size; S503, performing positional addition on the converted features; S504, the added features are subjected to three-layer convolution operations to obtain a high-resolution feature map X hf .

7. The method for panoramic image segmentation based on color and depth information according to claim 5, characterized in that: The processing of S6 includes: S601, for the high-resolution feature map X hf The same method is used to generate the weight head, adding the learnable relative position information and performing convolution operation to obtain X hf′ ; S602, for N s Background weight and M object weights Respectively with X hf′ Perform convolution operation to get N s +M prediction images, perform bilinear difference upsampling and threshold operation on the prediction images to obtain binary prediction images of the same size as the original images; S603, processing the conflicting pixels in each prediction image, and marking the pixels to which no type is assigned as background class.

8. The method for panoramic image segmentation based on color and depth information according to claim 7, characterized in that: The N obtained in S602 s +M prediction images, each of which corresponds to an instance or a background in the image.

Images