Intelligent driving multi-instance object segmentation method based on depth recognition framework

Abstract

The invention discloses an intelligent driving multi-instance object segmentation method based on a depth recognition framework. An FCN-8s network model structure is adopted to predict an object semantic category, and FCN-8s outputs a pixel-level probability value of an object instance. And an R-CNN network model is adopted to predict an object bounding box, and object position identification is completed. And clustering the predicted bounding box by using a Gaussian mixture model (GMM), wherein each component of the GMM corresponds to one instance. An optimal number of object instances is selected using Akaike Information Criterion (AIC). A conditional random field (CRF) model based on the ROI is established to solve the multi-object marking problem. And embedding a probability value obtained according to the depth recognition framework as single item energy of the CRF model, and embedding a difference energy loss item between two superpixels as binary item energy of the CRF model. And optimizing the conditional random field model by using a graph cut algorithm to finally obtain marks of a plurality of instances and realize segmentation of a multi-instance object.

Description

Technical Field

[0001] The present invention relates to a research point of image scene content understanding in the fields of computer vision, artificial intelligence, and image processing, and specifically to a multi-instance object segmentation method for intelligent driving based on a deep recognition framework. Background Art

[0002] Semantic segmentation is a fundamental problem in computer vision, which is crucial for overall scene understanding. Its main goal is to assign a unique category label to each pixel. Therefore, semantic segmentation is also called semantic labeling. In the past few decades, research on semantic segmentation has achieved impressive progress, but these works mainly deal with problems at the scene level. For overall scene understanding, scene-level category labeling is a preliminary understanding of the scene, without identifying and understanding individual objects and regions of individual objects, which cannot provide accurate information for further computer vision tasks such as image retrieval, image-based modeling, and intelligent driving. These tasks require further understanding of the scene, especially detailed information about the objects. For example, in an intelligent driving system, an unmanned vehicle needs to perceive the natural road environment and process visual information, accurately identify and distinguish individual objects in the scene, which helps to improve the safety and efficiency of intelligent driving.

[0003] In fact, the recognition and understanding of individual objects, such as object detection and object segmentation, have achieved encouraging results, but the semantic segmentation of multiple instances of objects in a scene is still insufficient. Compared with scene-level semantic segmentation, there is still a lot of room for improvement. Some work only focuses on the segmentation and extraction of object contours, while ignoring the semantic information of the object. The few works on object semantic segmentation mostly deal with situations where the object position and posture are very significant.

[0004] Due to the diversity of objects in texture, shape, posture, and occlusion, instance-level object segmentation remains a challenging task. In the era of deep learning, convolutional neural networks have significant advantages over traditional models in feature extraction and computing power, and are naturally introduced into the work of object instance segmentation. Existing work uses CNN networks to extract features and obtain powerful discriminative feature structures. However, the accuracy of instance-level object segmentation still needs to be improved. Summary of the invention

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A multi-instance object segmentation method for intelligent driving based on a deep recognition framework, comprising:

[0007] Step 1: Use the FCNs network model structure to predict the semantic category of the object. According to the instance benchmark tag information in the object dataset, the pre-trained open source model FCN-8s is fine-tuned, and FCN-8s outputs the pixel-level probability value of the object instance.

[0008] Step 2: Use the R-CNN network model to predict the object bounding box and complete the object location recognition. Specifically, a publicly available model pre-trained on the ILSVRC2013 dataset is used. The network structure extracts rich features of each candidate region and outputs its category score.

[0009] Step 3: Use Gaussian mixture model (GMM) to cluster the predicted bounding boxes, and each component of GMM corresponds to one instance. Use Akaike information criterion (AIC) to select the optimal number of object instances. The one with the smallest AIC value is selected as the preferred number of object instances to complete the identification of the number of object instances.

[0010] Step 4: Establish a conditional random field (CRF) model based on the region of interest (ROI) to solve the multi-object labeling problem. Use an existing mature algorithm to generate a superpixel set and convert the obtained pixel-level probability value into a superpixel-level probability value. The probability value obtained according to the deep recognition framework is embedded as a single energy term in the CRF model, and the difference energy loss term between two superpixels is embedded as a binary energy term in the CRF model.

[0011] Step 5: Use the graph cut algorithm to optimize the conditional random field model, and finally obtain the labeling of multiple instances to achieve the segmentation of multiple instance objects in the intelligent driving scenario.

[0012] Preferably, in the method for multi-instance object segmentation of intelligent driving based on a deep recognition framework, in step 1, the pre-trained open source model FCN-8s is fine-tuned, and the output number of the fully connected layer and the scoring layer of the model is set to the number of object categories in the data set. Object category refers to the object instance label, including the void category. In addition, the FCN-8s structure is also used to predict the semantic category probability value of the object.

[0013] Preferably, in the method for multi-instance object segmentation of intelligent driving based on a deep recognition framework, in step 2, the candidate regions are generated using a selective search algorithm, and the R-CNN network model is used to extract rich features of each candidate region, and its category score is output. For each object category, the candidate regions are sorted according to the category score, and the first m regions are selected to form an initial object bounding box set. The value of m varies from 1 to 50. The features of each bounding box include its normalized center coordinates, width, and height.

[0014] Preferably, in the method for multi-instance object segmentation of intelligent driving based on a deep recognition framework, in step 3, after clustering the predicted bounding boxes, each bounding box divides a corresponding object instance. Since there is an offset between the center of the cluster component and the center of the real object instance, it is necessary to relocate the center of each clustered object instance. For each pixel p, the frequency of its appearance in the bounding box set of the object is counted, and the range formed by the pixel set that satisfies the frequency greater than the set threshold is used as the location of the final bounding box of the object instance.

[0015] Preferably, in the method for multi-instance object segmentation of intelligent driving based on a deep recognition framework, in step 4, a single item of the CRF model comprises an object probability value, a shape probability value and a position probability value. The object probability maps the pixel-level object probability output by FCN-8s to the superpixel level, which means the possibility of each superpixel being assigned to each object instance. Objects usually have typical shape features, so the shape probability is embedded in the energy formula as prior information of the object. The detection of object instances implies the possibility of the object's location, and the position probability of each superpixel is generated based on the position of the bounding box and the superpixel position indication.

[0016] Preferably, the shape probability value uses centripetal force to describe the shape. In the manifold space of object features, each pixel belonging to the same object is subject to the centripetal force toward the spatial centroid of the object. The closer to the spatial centroid, the greater the centripetal force it receives, and the greater the probability that it belongs to the object. Shape probability includes three aspects: angular distance, physical distance, and color distance. Angular distance measures the angle between the superpixel and its centroid in the polar coordinate system. Physical distance is measured in polar radius, and color distance is measured in RGB color space. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the method framework of the present invention. DETAILED DESCRIPTION

[0018] The present invention is described in further detail below so that those skilled in the art can implement it according to the description.

[0019] The present invention provides a method for intelligent driving multi-instance object segmentation based on a deep recognition framework, which can obtain segmentation results of multiple objects in an image scene based on deep recognition. The overall process is as follows:

[0020] First, a deep recognition framework is used to perform recognition analysis on the scene, which includes semantic category recognition, object location recognition, and instance quantity recognition.

[0021] The FCNs network model structure is used to predict the semantic category of the object. Specifically, the open source model fcn8s-heavy-pascal.caffemodel (FCN-8s for short) pre-trained on the PASCAL dataset is fine-tuned according to the object instance benchmark tag information that has been annotated for training in the object dataset, and FCN-8s outputs the pixel-level probability value of the object instance. The R-CNN network model is used to predict the object bounding box and complete the object location recognition. Specifically, the bvlc_reference_ilsvrc13.caffemodel (R-CNN for short) pre-trained on the ILSVRC2013 dataset is used. The network structure extracts rich features of each candidate region and outputs its category score. The predicted bounding box is clustered using a Gaussian Mixture Model (GMM), and each component of the GMM corresponds to an instance. The Akaike Information Criterion (AIC) is used to select the optimal number of object instances. The object instance number with the smallest AIC value is selected as the preferred object instance number to complete the recognition of the object instance number.

[0022] Then, a Conditional Random Field (CRF) model based on the Region Of Interest (ROI) is established to solve the problem of multi-object labeling. A mature algorithm is used to generate a superpixel set, and the obtained pixel-level probability value is converted into a superpixel-level probability value. The probability value obtained according to the deep recognition framework is embedded as a single energy term of the CRF model, and the difference energy loss term between the two superpixels is embedded as a binary energy term of the CRF model. Using the graph cut algorithm, optimization is performed on the conditional random field model, and finally multiple instance labels are obtained to achieve the segmentation of multi-instance objects.

[0023] Based on the above process, the Deep Recognition Framework (DRF) predicts object probabilities, object bounding boxes, and the number of object instances. DRF uses a common deep learning algorithm to generate rough object tags and bounding box probability suggestions, and then gives a deep understanding of the object.

[0024] FCNs introduced convolutional neural networks (CNNs) into semantic segmentation and achieved exciting results. Multi-instance object segmentation can be regarded as a type of semantic segmentation. On the basis of identifying semantic categories, multiple object instances of the same semantic category are labeled, that is, one label corresponds to one object instance. The FCNs network model structure was adopted, and the open source model fcn8s-heavy-pascal.caffemodel (FCN-8s) pre-trained on the PASCAL dataset was selected. Then, the model was fine-tuned according to the instance benchmark labeling information (groundtruth) in the object dataset, and the output number of the fully connected layer and the scoring layer was set to 8, that is, the output object categories were 8, including the void category, which represents the background area. The output of FCN-8s provides pixel-level probability values ​​for 8 object instances. In addition, the FCN-8s structure is also used to predict the semantic category probability of the object, such as the probability of belonging to the categories of horse, person, car, etc.

[0025] R-CNN is a classic model in the field of object recognition. The R-CNN network model is used to predict object bounding boxes. Specifically, a publicly available model pre-trained on the ILSVRC2013 dataset is used. The network structure extracts rich features of each candidate region (the candidate region is generated by the selective search algorithm) and outputs its category score. For each object category, such as the car category, the candidate regions are sorted according to the category score, and the top m candidate regions are selected to form the initial object bounding box set, denoted as B. The value of m is in the range of 1 to 50. Each object bounding box can be represented as (min(x), min(y), max(x), max(y), score), where min(x) and min(y) are the coordinates of the upper left corner of the bounding box, max(x) and max(y) are the coordinates of the lower right corner of the bounding box, and score is the score of the bounding box.

[0026] When identifying object categories, there are cases where multiple object instances of the same category are identified as a whole category. When identifying object bounding boxes, there are cases where the bounding boxes of multiple object instances overlap or cover each other, which sometimes interferes with the precise positioning of object instances. When the number of object instances is known, redundancy can be effectively eliminated and the accuracy of object instance segmentation can be improved. Therefore, in order to accurately segment each object instance, it is necessary to determine the exact number of object instances within the value range of m.

[0027] In this section, we use Gaussian mixture model (GMM) to cluster the bounding boxes in the initial object bounding box set B. The features of each bounding box include its normalized center coordinates, width, and height. Each component of the clustered GMM corresponds to an object instance. The Akaike Information Criterion (AIC) is used to select the number of clusters with the minimum AIC value as the optimal number of instances, denoted as k.

[0028] After determining the optimal number of instances, each bounding box in the object bounding box set B can be divided into a corresponding object instance, but sometimes there is an offset between the center of the cluster component and the center of the real object instance, so it is necessary to relocate the center of each clustered object instance. For an object instance s, the bounding box corresponding to it after clustering is represented as B s (a subset of B), and denote its final bounding box as b s For each pixel p, count the number of pixels in B s The frequency of appearance in the bounding box of is denoted as A high frequency means that the pixel appears in most bounding box areas, and the pixel is more likely to belong to a real object instance. The set of all pixels of , denoted as P. The final object instance bounding box b s The positioning is (min(P x ),min(P y ),max(P x ),max(P y )), where min(P x )、min(P y ) is the minimum value of the pixel set in the x-axis and y-axis directions, corresponding to b s The upper left corner; max(P x )、max(P y ) is the maximum value of the pixel set in the x-axis and y-axis, corresponding to b s The lower right corner, threshold τ s Take the value as 0.5. In this way, B s The multiple bounding boxes in are converted into the final object instance bounding box b s .

[0029] like Figure 1 As shown, the bounding box in the prediction information is the detection output of the CNN network model when the number of object instances is unknown. The object instance number prediction is the improved bounding box under a fixed number. The figure shows two object instances.

[0030] After the DRF deep recognition framework gives the initial object region, a CRF model based on the region of interest is established to solve the multi-object labeling problem. For each image I, this method uses the TurboPixels algorithm to generate thousands of superpixels. The definition of the energy function is shown in formula (1), where l i represents the object label of superpixel i, L represents the set of all object labels, ranging from 0 to k. U(·) and V(·) are the single term and binary term in the energy function, respectively, and R is the region of interest.

[0031]

[0032] For superpixel i, U i (l i ) is to mark the object l i The energy loss term assigned to i is calculated as shown in Formula 2. c is the object probability value, P f is the shape probability value, P b is the position probability value, α c , α f , α b The corresponding object weight parameter, shape weight parameter and position weight parameter are 0.3, 0.1 and 1.5 respectively. For specific object labels, l i The value range is [0,...,k], where 0 corresponds to the background area. i When the background area is marked, β is 1, otherwise it is 0.

[0033]

[0034] The object probability value is calculated by mapping the pixel-level probability value of the object instance output by FCN-8s to the superpixel level, which means the possibility that each superpixel belongs to each object instance.

[0035] Objects usually have typical shape features, and different objects can be effectively distinguished by their shapes. Therefore, shape probability can be embedded in the energy formula as a priori information. Here, centripetal force is used to describe the shape. In the manifold space of object features, each pixel belonging to the same object is subject to the centripetal force toward the spatial center of mass of the object. The closer the distance is to the spatial center of mass, the greater the centripetal force it receives, and the greater the probability that it belongs to the object. Shape probability includes three aspects: angular distance, physical distance, and color distance. Angular distance measures the angle between superpixel i and the center of mass of the object bounding box in the polar coordinate system, that is, the polar coordinate angle. The polar radius is used as the physical distance, and the color distance is the distance in the RGB space. The probability density functions of angle, physics, and color are then estimated, and the angle probability P of each superpixel is predicted. ang, physical distance probability P dis , color distance probability P clr The shape probability is calculated as:

[0036] P f (i,l i )=P ang (i,l i ) dis (i,l i ) clr (i,l i ) (Formula 3)

[0037] The detection of object instances implies the possibility of the object location. The location probability of each superpixel is generated based on the location of the bounding box and the superpixel location indication. The location probability is calculated as shown in Formula 4. b (i,l i ) indicates that the current superpixel i belongs to the current assigned object label l i The location probability value is Indicates whether superpixel i is located in object l i If it is in the bounding box of l, it takes the value 1, otherwise it takes the value 0. i The ratio of the number of pixels of superpixel i in the bounding box of .

[0038]

[0039] V in Formula 1 i,j (l i ,l j ) is the difference energy loss term between two superpixels i and j, which is calculated as follows: smo(i,j) represents the smoothing loss between adjacent superpixels, dsm(i,j) represents the difference loss between non-adjacent superpixels, l smo and l dis are the corresponding smoothing weight parameters and difference weight parameters, which are 1 and 5 respectively. ω(i,j) represents the adjacency relationship between two superpixels, which is 1 if they are adjacent and 0 otherwise. The differences between adjacent superpixels are embedded in the smoothing loss, including color and boundary features. Color features reflect appearance differences, and boundaries reflect texture differences. The smoothing term is a linear combination of color and boundary features. The difference loss measures the loss of assigning object labels to non-adjacent superpixels, including color and distance. For two non-adjacent superpixels, obvious differences in color or distance will increase the loss.

[0040] V i,j (l i ,l j )=λ smo ω(i,j)exp(smo(i,j))+λdis (1-ω(i,j))exp(dsm(i,j)) (Formula 5)

[0041] After constructing the conditional random field model, the graph cut algorithm is used for optimization, and finally the labels of multiple object instances are obtained to achieve the segmentation of multiple instance objects in the intelligent driving scenario.

[0042] The dataset used in this method is the public dataset TUD provided by the Technical University of Darmstadt in Germany. The dataset is a road scene and contains 1,216 pedestrian object instance samples. Pedestrians have different postures and clothing, and there is occlusion, which increases the difficulty of object instance segmentation. This method expands the annotation of semantic categories such as cars, ground, and buildings on this dataset, as well as the labeling information of each instance object of the car category. Experiments have shown that compared with the object instance segmentation method with the highest accuracy on this dataset, this method has improved the accuracy by nearly 8 percentage points, proving the effectiveness of this method and its applicability to intelligent driving scenarios.

[0043] In the future, we will further improve the accuracy of the model, further optimize the algorithm, and better improve the accuracy of object instance segmentation, thereby improving the safety of intelligent driving applications.

Claims

1. A multi-instance object segmentation method for intelligent driving based on a deep recognition framework, characterized in that: Include: Step 1: Use the FCNs network model structure to predict the semantic category of the object; according to the instance benchmark tag information in the object dataset, the open source model FCN-8s outputs the pixel-level probability value of the object instance; Step 2: Use the R-CNN network model to predict the object bounding box and complete the object location recognition; specifically, a publicly available model pre-trained on the ILSVRC2013 dataset is used. The network structure extracts rich features of each candidate region and outputs its category score; Step 3: Use Gaussian mixture model (GMM) to cluster the predicted bounding boxes, and each component of GMM corresponds to an instance; use Akaike information criterion (AIC) to select the optimal number of object instances; the one with the smallest AIC value is selected as the preferred number of object instances to complete the identification of the number of object instances; Step 4: Establish a conditional random field (CRF) model based on the region of interest (ROI) to solve the multi-object labeling problem; use an existing mature algorithm to generate a superpixel set, and convert the obtained pixel-level probability value into a superpixel-level probability value; embed the probability value obtained according to the deep recognition framework as a single energy embedding of the CRF model, and embed the difference energy loss term between two superpixels as a binary energy embedding of the CRF model; Step 5: Use the graph cut algorithm to optimize the conditional random field model, and finally obtain the labeling of multiple instances to achieve the segmentation of multiple instance objects in the intelligent driving scenario.

2. The method for segmenting multiple instances of intelligent driving objects based on a deep recognition framework according to claim 1, characterized in that: In step 1, the pre-trained open source model FCN-8s is fine-tuned, and the output numbers of the fully connected layer and the scoring layer of the model are set to the number of object categories in the dataset; the object category refers to the object instance label, including the void category; in addition, the FCN-8s structure is used to predict the semantic category probability value of the object.

3. The method for segmenting multiple instances of objects in intelligent driving based on a deep recognition framework according to claim 1, characterized in that: In step 2, candidate regions are generated using a selective search algorithm, and the R-CNN network model is used to extract rich features of each candidate region and output its category score; for each object category, the candidate regions are sorted according to the category score, and the first m regions are selected to form an initial object bounding box set; the value of m varies from 1 to 50 and is automatically generated by the algorithm; the features of each bounding box include its normalized center coordinates, width, and height.

4. The method for segmenting multiple instances of objects in intelligent driving based on a deep recognition framework according to claim 1, characterized in that: In step 3, after the predicted bounding boxes are clustered, each bounding box divides a corresponding object instance. Since there is an offset between the center of the cluster component and the center of the real object instance, the center of each clustered object instance needs to be relocated; for each pixel p, its frequency of appearance in the object's bounding box set is counted, and the range formed by the set of pixels that satisfy the frequency greater than the set threshold is used as the location of the final bounding box of the object instance.

5. The method for segmenting multiple instances of objects in intelligent driving based on a deep recognition framework according to claim 1, characterized in that: In step 4, a single item of the CRF model includes an object probability value, a shape probability value, and a position probability value; the object probability maps the pixel-level object probability output by FCN-8s to the superpixel level, which means the possibility of assigning each superpixel to each object instance.

6. The method for segmenting multiple instances of intelligent driving objects based on a deep recognition framework according to claim 1, characterized in that: The centripetal force is used to describe the shape; in the manifold space of object features, each pixel belonging to the same object is acted upon by the centripetal force toward the spatial centroid of the object; the closer it is to the spatial centroid, the greater the centripetal force it receives and the greater the probability that it belongs to the object; the shape probability includes three aspects: angular distance, physical distance, and color distance; the angular distance measures the angle between the superpixel and its centroid in the polar coordinate system; the physical distance is measured by the polar radius, and the color distance is measured in the RGB color space.

7. The method for segmenting multiple instances of objects in intelligent driving based on a deep recognition framework according to claim 1, characterized in that: After the DRF deep recognition framework gives the initial object region, a CRF model based on the region of interest is established. For each image I, the TurboPixels algorithm is used to generate thousands of superpixels; The definition of energy function is shown in formula (1), where l i represents the object label of superpixel i, L represents the set of all object labels, ranging from 0 to k; U(·) and V(·) are the single term and binary term in the energy function, respectively, and R is the region of interest; For superpixel i, U i (l i ) is to mark the object l i The energy loss term assigned to i is calculated as shown in Formula 2; P c is the object probability value, P f is the shape probability value, P b is the position probability value, α c , α f , α b They are the corresponding object weight parameter, shape weight parameter and position weight parameter, which are 0.3, 0.1 and 1.5 respectively; for the specific object label, l i The value range is [0,...,k], 0 corresponds to the background area; when l i When marking the background area, β is 1, and 0 in other cases; U i (l i ) = α c exp(-P c (i, l i )) + α f exp(-P f (i, l i )) + α b (1 - β)exp(-P b (i, l i )) (Formula 2) The object probability value is calculated by mapping the pixel-level probability value of the object instance output by FCN-8s to the superpixel level, which means the probability that each superpixel belongs to each object instance; The centripetal force is used to describe the shape; in the manifold space of object features, each pixel belonging to the same object is subject to the centripetal force toward the object's spatial centroid; the closer to the spatial centroid, the greater the centripetal force it receives, and the greater the probability of its belonging to the object; the shape probability includes three aspects: angular distance, physical distance, and color distance; the angular distance measures the angle between the superpixel i and the object's bounding box centroid in the polar coordinate system, that is, the polar coordinate angle; the polar radius is the physical distance, and the color distance is the distance in the RGB space; then the probability density functions of angle, physics, and color are estimated, and the angle probability P of each superpixel is predicted ang , physical distance probability P dis , color distance probability P clr ; The shape probability is calculated as: P f (i, l i ) = P ang (i, l i ) P dis (i, l i ) P clr (i, l i ) (Formula 3) The detection of object instances implies the possibility of object locations. The location probability of each superpixel is generated based on the location of the bounding box and the superpixel location indication; The calculation method of position probability is shown in Formula 4; P b (i,l i ) indicates that the current superpixel i belongs to the current assigned object label l i The location probability value is Indicates whether superpixel i is located in object l i If it is in the bounding box, it takes the value 1, otherwise it takes the value 0; y means in l i The ratio of the number of pixels of superpixel i in the bounding box of ; V in Formula 1 i,j (l i ,l j ) is the difference energy loss term between two superpixels i and j, which is calculated as follows: smo(i,j) represents the smoothing loss between adjacent superpixels, dsm(i,j) represents the difference loss between non-adjacent superpixels, l smo and l dis are the corresponding smoothing weight parameters and difference weight parameters, which are 1 and 5 respectively; ω(i,j) represents the adjacency relationship between two superpixels, which is 1 if they are adjacent and 0 otherwise; the difference between adjacent superpixels is embedded in the smoothing loss, including color and boundary features; Color features reflect appearance differences, and boundaries reflect texture differences. The smoothness term is a linear combination of color and boundary features. The difference loss measures the loss of assigning object labels to non-adjacent superpixels, including both color and distance. For two non-adjacent superpixels, obvious differences in color or distance will increase the loss. V i,j (l i ,l j ) = λ smo ω(i,j) exp(smo(i,j)) + λ dis (1 - ω(i,j)) exp(dsm(i,j)) (Equation 5) After constructing the conditional random field model, the graph cut algorithm is used for optimization, and finally the labels of multiple object instances are obtained to achieve the segmentation of multiple instance objects in the intelligent driving scenario.

Images