An unsupervised monocular three-dimensional target detection method based on video sequence and pre-training instance segmentation

Abstract

The application provides an unsupervised monocular three-dimensional target detection method based on a video sequence and a pre-trained instance segmentation. The main steps are as follows: a camera with a known internal parameter is used to shoot a video sequence frame of a certain length in a certain scene, a monocular depth network is self-supervisedly trained by using the projection relationship between the sequence images, and the monocular depth estimation value of the sequence images is learned; then, an instance segmentation result of the image is directly predicted by using a pre-trained instance segmentation network, the instance segmentation result is combined with the camera internal parameter and the learned monocular depth estimation value, and is back-projected into a three-dimensional space to obtain pseudo-radar data of each instance; finally, an unsupervised clustering method is used to filter out outliers, a minimum circumscribed rectangle of the point set is solved in a bird's eye view (x-z plane), a maximum-minimum height difference of the point set is solved in a y-axis direction, and finally a three-dimensional target detection bounding box of the object is obtained. The application can complete three-dimensional target detection of any new scene without manual annotation by using the sequence information of the front and rear frames of the video and the pre-trained instance segmentation network, and can significantly reduce the manual annotation cost required for learning three-dimensional target detection of a new scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of target detection, and proposes an unsupervised monocular three-dimensional target detection method based on video sequences and pre-trained instance segmentation. BACKGROUND

[0002] With the development of deep learning, many computer vision related tasks have broken through the limitations of traditional methods and achieved breakthrough development, for example, in the emerging field of autonomous driving, relying on the high accuracy and reliability of upstream tasks such as target detection and scene semantic understanding, people can try to complete the related control of autonomous driving through means such as three-dimensional target detection. According to the form of sensor input data, the three-dimensional target detection task can be roughly divided into three categories: laser radar type, multi-view type and monocular type. Among them, the monocular three-dimensional target detection task refers to the detection of targets in three-dimensional space in the image combined with the parameters of the camera under the condition of only monocular image as data input.

[0003] Among these forms, although the laser radar has high precision, the instrument is expensive and the detection distance is limited, and it often needs to be combined with a multi-view camera to complete the construction of an autonomous driving sensing system, for example, today's TESLA, Baidu Apollo and other enterprises have already had a relatively mature multi-view combined with laser radar solution. However, if an autonomous driving system can rely only on a monocular camera to complete the three-dimensional detection of targets, it will greatly save costs and have extremely high promotional value, which has become the focus and research hotspot of current three-dimensional target detection.

[0004] However, monocular three-dimensional target detection has the following challenges: (1) monocular images contain less information, only color information of three channels of RGB, lack of spatial depth information that can be obtained by laser radar or multi-view images, how to estimate the distance of the object and the depth of the pixel point will be the key to this problem; (2) the existing three-dimensional target detection public data set is usually oriented to the field of autonomous driving, and the style of the image is limited to foreign rural and urban areas, so the generalization performance of the trained model cannot be effectively guaranteed when it is applied to a new scene; (3) obtaining a real three-dimensional detection bounding box of an object usually needs to obtain its laser radar information first, and then hire professional annotators to annotate the object in the radar scene, which requires a large cost. At present, it is still a great challenge to develop an unsupervised method that can obtain a three-dimensional detection bounding box in a new scene without labeling.

[0005] In order to effectively improve the accuracy of monocular three-dimensional target detection, the present method mines more information based on the sequence characteristics of video, and jointly constructs a three-dimensional target detection network without label information and with good generalization by means of a pre-trained instance segmentation network model. SUMMARY

[0006] In order to effectively reduce the high labeling cost of three-dimensional target detection task in a new scene, the application can unsupervisedly generate the three-dimensional target detection bounding box of the image by using the motion information between the video continuous frames and the pre-trained instance segmentation network in other data sets.

[0007] To achieve the above purpose, the technical solution adopted by the application is as follows:

[0008] An unsupervised monocular three-dimensional target detection method based on video sequence and pre-trained instance segmentation, characterized by comprising the following steps:

[0009] (1) Obtain the camera intrinsic matrix;

[0010] (2) Use the camera to shoot a certain length of video sequence frames in a certain scene;

[0011] (3) Through the self-supervised monocular depth network, the projection relationship between the sequence images is used to train and learn self-supervision to obtain the monocular depth estimation value of the sequence images;

[0012] (4) The pre-trained instance segmentation network is used to directly predict the instance segmentation result of the image;

[0013] (5) The obtained instance segmentation result is combined with the monocular depth estimation value learned in step (3) and the camera intrinsic matrix obtained in step (1) to be back projected into the three-dimensional space to obtain the three-dimensional point set of each instance region;

[0014] (6) Use an unsupervised clustering method to cluster the three-dimensional point set of each instance region, further filter out outliers, obtain the three-dimensional point set of the instance main body, and then solve the minimum circumscribed rectangle of the three-dimensional point set of the instance main body in the BEV (Bird's Eye View, bird's eye view) to obtain the two-dimensional bounding box of the x-z plane, and calculate the maximum-minimum height difference of the y-axis direction of the three-dimensional point set of the instance main body. The maximum-minimum height difference in the y-axis direction is taken as the y-axis height, and finally the three-dimensional detection bounding box of the object is obtained.

[0015] As described above, the method is characterized in that in step (3), the monocular depth estimation value of the sequence image is obtained by using the projection relationship between the sequence images through the self-supervised monocular depth network, and the specific process is as follows: first, predict the monocular depth estimation value and the pose transformation matrix of the current frame and the adjacent frame, then perform pose transformation on the monocular depth estimation value of the adjacent frame to obtain the monocular depth estimation value of the adjacent frame after the pose change, and calculate the loss of the monocular depth estimation value of the adjacent frame and the current frame to realize the self-supervised training and learning.

[0016] The method as described above, characterized in that the pre-training instance segmentation network used in step (4) should use a dataset containing the target classes required for the three-dimensional target detection task in its pre-training process and have good generalization performance.

[0017] The method as described above, characterized in that the specific process of the instance segmentation result in step (5) combined with the monocular depth estimation value and the camera intrinsic matrix back projection is: converting the pixel point p after instance segmentation into homogeneous coordinate form p = (u, v, 1), wherein u and v are the horizontal and vertical coordinates of the pixel point in the two-dimensional image in pixel units; combining the monocular depth estimation d p = D t (p) obtained in step (3) with the camera intrinsic matrix K obtained in step (1) to calculate the coordinates (U, V, W, 1) T = K -1 d p (u, v, 1) T of the pixel point p in the three-dimensional scene, wherein U, V, and W respectively correspond to the coordinate values on the x, y, and z axes of the camera coordinate system, which are in meters, and finally obtain the three-dimensional point set of each instance region.

[0018] The method as described above, characterized in that the specific process of converting the three-dimensional point set of each instance region into the three-dimensional point set of the instance main body and further calculating the three-dimensional bounding box in step (6) is: first applying an unsupervised clustering algorithm to the three-dimensional point set of each instance region to obtain multiple clustering clusters; taking the clustering cluster with the most points as the three-dimensional point set of the instance main body, and removing the points contained in the other clustering clusters as outliers; based on the y-axis direction, obtaining the maximum and minimum values of the three-dimensional point set of the instance main body, and taking the difference between the two as the y-axis height of the three-dimensional bounding box, and simultaneously solving the minimum circumscribed rectangle on the x-z plane corresponding to the three-dimensional point set of the instance main body, and finally obtaining the three-dimensional bounding box.

[0019] The method as described above, characterized in that: for each adjacent frame of the current frame, preferably the next frame of the current frame, the self-supervised monocular depth estimation network, preferably the sfm-learner, Monodepth series, sc-depth or packnet network structure, wherein the network for pose estimation is preferably a ResNet-18 network with the input layer channel number modified to 6.

[0020] The method as described above, characterized in that: the pre-training dataset of the instance segmentation network is preferably the MSCOCO dataset, and the network is preferably the Mask R-CNN network structure; for the result obtained after instance segmentation, further preferably the pixel region with a class confidence greater than or equal to 0.5 and a bounding box internal pixel confidence greater than or equal to 0.5, to obtain the final instance segmentation result.

[0021] The method described above is characterized in that: for each instance of the region three-dimensional point set, an unsupervised clustering algorithm is used, preferably a DBSCAN algorithm, the neighborhood radius E in the algorithm parameters is preferably 0.8 meters, and the minimum number of samples in the neighborhood MinPts is preferably 10; for the category that meets the object vertical to the ground assumption, the instance main three-dimensional point set is solved in the x-z plane to obtain the minimum circumscribed matrix, and a rotation jamming algorithm is preferably used. BRIEF DESCRIPTION OF DRAWINGS

[0022] The application will be further described below in combination with the drawings and specific embodiments.

[0023] Figure 1 A flowchart of the unsupervised monocular three-dimensional target detection method based on video sequences and pre-trained instance segmentation of the application;

[0024] Figure 2 A segmentation effect diagram of the pre-trained instance segmentation network directly migrated to the vehicle category in an embodiment. DETAILED DESCRIPTION

[0025] As mentioned above, the application proposes an unsupervised monocular three-dimensional target detection method based on video sequences and pre-trained instance segmentation, and the specific embodiments of the application will be described below in combination with the drawings. The specific neural network used in the embodiments is only for reference and is not limited to this network. As long as the task function can be achieved, other replacements can be made.

[0026] As Figure 1 shown, the unsupervised monocular three-dimensional target detection method based on video sequences and pre-trained instance segmentation of the application includes the following steps in one embodiment:

[0027] (1) First, obtain the intrinsic matrix K of the camera, and use the camera to shoot a certain length of video sequence frames in a certain scene. The camera support can be simply used to walk and shoot, or the camera can be fixed on a vehicle to shoot the scene at a low speed, thereby obtaining a video image sequence in a certain scene.

[0028] (2) Through a self-supervised monocular depth network, the projection relationship between the sequence images is used to self-supervise the learning of monocular depth estimation of the sequence images. In this embodiment, monodepth2 network is used as a self-supervised monocular depth estimation prediction network, and other embodiments can also be replaced as long as the effect is guaranteed. Taking the KITTI public dataset as an example, the data is an automatic driving scene in a foreign rural environment, and a certain length of video sequence is shot in each scene. Select one of the scenes as the video sequence of this embodiment, for example, the scene numbered 2011_10_03_drive_0047, which contains 837 frames of images.

[0029] In each epoch, all frames except the last one are taken as the training set, and each frame I t (hereinafter referred to as the current frame) and the next frame I t′ (hereinafter referred to as the adjacent frame) are taken as inputs for training, and the process is as follows:

[0030] (2.1) The current frame and the adjacent frame are respectively input into a monocular depth network with a U-Net type Encoder-Decoder structure to obtain the depth estimates D t and D t′ of the current frame and the adjacent frame.

[0031] (2.2) The current frame and the adjacent frame are spliced to obtain a HxWx6 vector, where H and W are the height and width of the frame; the vector is input into a pose estimation network, and in this embodiment, a ResNet-18 network is selected and the channel number of the input layer is modified to 6 to adapt to the vector dimension after splicing, thereby obtaining the pose transformation matrix (i.e. homography matrix) H from the adjacent frame to the current frame; for the adjacent frame pixel point i', its homogeneous coordinate form p i′ = (x, y, 1) T , then its corresponding pixel point i in the current frame has coordinates p i = Hp i′ , and the vector values of the two on the image level should have a corresponding relationship, so the L1 loss is used to iteratively train the pose transformation estimation network.

[0032] (2.3) For the adjacent frame pixel point i' and its corresponding pixel point i in the current frame, their respective monocular depth estimates should also have a corresponding relationship, so a smooth L1 loss is used for the two, where D(p) represents the monocular depth estimate value of a pixel point p on the monocular depth estimate map D, so as to iteratively train the monocular depth estimation network.

[0033] After 15 epochs, the training loss is basically converged, and the training of the monocular depth estimation network is completed. In the subsequent network inference process, only the parameters of the monocular depth estimation network are needed to complete the depth estimation of any image.

[0034] (3) The instance segmentation information of the current frame image is predicted by a pre-trained instance segmentation network. In this embodiment, a Mask R-CNN network pre-trained on the MSCOCO dataset is used as the instance segmentation network, and other embodiments can also be replaced under the premise of ensuring the effect.

[0035] The network is pre-trained on the MSCOCO dataset and can well complete the detection of objects such as vehicles, pedestrians, and bicycles, meeting the task requirements of the method. Thus, we can complete the prediction of instance segmentation information for each frame of the photographed video. In the embodiment, the object class confidence is set to 0.5, and the mask pixel confidence is set to 0.5, and a relatively accurate instance segmentation can be obtained, as shown in Figure 2 Fig. 3 shows the instance segmentation prediction obtained under the above conditions.

[0036] (4) The instance segmentation prediction obtained in step (3) is back projected into the three-dimensional space through the intrinsic matrix of the camera and the corresponding monocular depth estimation. Denote the point set obtained by segmentation as P, and take the homogeneous coordinates of the pixel point p e P as p = (u, v, 1). Through the intrinsic matrix K of the camera and the monocular depth estimation d p = D t (p) obtained in step (3), the coordinates (U, V, W, 1) T = K -1 d p p T of the pixel point in the three-dimensional scene under the camera coordinate system are calculated according to the pinhole camera model, where U, V, and W respectively correspond to the values on the x, y, and z axes of the camera coordinate system.

[0037] Thus, the pixel points in the point set P can be one-to-one mapped to their coordinates in the three-dimensional space, and the new point set is denoted as the regional three-dimensional point set, which is also referred to as pseudo-radar (point cloud) information in the field.

[0038] (5) For the regional three-dimensional point set, the DBSCAN algorithm is further selected in the embodiment to extract the main part of the instance and filter out outliers. The specific process is as follows:

[0039] (5.1) Set the parameters required by the DBSCAN algorithm: the minimum distance of the same class is 0.8 m, and the minimum number of samples in the neighborhood is 10.

[0040] (5.2) After the DBSCAN algorithm, a plurality of class clusters are output, and the cluster with the largest number of samples is selected as the main part of the instance, which is referred to as the instance main three-dimensional point set; the sample points of the remaining clusters are regarded as outliers and discarded. In the pseudo-radar information after filtering out the outliers, the red points are outliers, and the blue points are the retained instance main three-dimensional point set, which is used for further bounding box solving. Figure 1

[0041] ​(6) Based on the basic assumption that the object is perpendicular to the road surface, the x and z coordinates of the instance object three-dimensional point set are taken, and the minimum circumscribed rectangle is solved on the x-z plane by using the rotating calipers algorithm, as the length and width of the three-dimensional bounding box; the maximum and minimum values of the entire point set in the y-axis direction are taken, and the difference is taken as the height of the three-dimensional bounding box, and finally combined into the three-dimensional detection bounding box of the object.

[0042] Although the present application is described above in conjunction with the drawings, the present application is not limited to the specific embodiments described above, and any changes that can be conceived by those skilled in the art without departing from the spirit of the present application should fall within the scope of the present application. In addition, parts of the present application that are not described in detail are known in the art.

Claims

1. An unsupervised monocular three-dimensional object detection method based on video sequences and pre-trained instance segmentation, characterized in that, The method comprises the following steps: (1) obtaining a camera intrinsic matrix; (2) using the camera, shooting a certain length of video sequence frames in a certain scene; (3) through a self-supervised monocular depth network, using the projection relationship between the sequence images, self-supervised training and learning are carried out to obtain monocular depth estimation values of the sequence images; (4) through a pre-trained instance segmentation network, the instance segmentation result of the image is directly predicted; (5) combining the obtained instance segmentation result, the monocular depth estimation value learned in step (3) and the camera intrinsic matrix obtained in step (1), the monocular depth estimation value is back-projected into a three-dimensional space to obtain the coordinate values of the pixel points in the three-dimensional space after instance segmentation, and the coordinate values of the pixel points in the three-dimensional space contained by each instance are combined to obtain a region three-dimensional point set of each instance; (6) using an unsupervised clustering method, the region three-dimensional point set of each instance is clustered to further remove outliers to obtain an instance main three-dimensional point set, and then a minimum circumscribed rectangle of the instance main three-dimensional point set is solved in a bird's eye view to obtain a two-dimensional bounding box of the x-z plane and the maximum-minimum height difference of the instance main three-dimensional point set in the y-axis direction, and the maximum-minimum height difference in the y-axis direction is taken as the y-axis height to finally obtain a three-dimensional detection bounding box of the object.

2. The unsupervised monocular 3D object detection method based on video sequence and pre-trained instance segmentation of claim 1, wherein In step (3), the monocular depth estimation values of the sequence images are obtained through the self-supervised monocular depth network using the projection relationship between the sequence images, and the specific process is as follows: the monocular depth estimation values and the pose transformation matrix of the current frame and the adjacent frame are predicted first, then the monocular depth estimation value of the adjacent frame is subjected to pose transformation to obtain the monocular depth estimation value of the adjacent frame after the pose change, and the monocular depth estimation value of the adjacent frame after the pose change is combined with the monocular depth estimation value of the current frame to calculate the loss, so as to realize the self-supervised training and learning.

3. The unsupervised monocular 3D object detection method based on video sequence and pre-trained instance segmentation of claim 1, wherein In step (4), the pre-trained instance segmentation network used should use a data set containing the detection target categories required by the three-dimensional target detection task in the pre-training process and have good generalization performance.

4. The unsupervised monocular 3D object detection method based on video sequence and pre-trained instance segmentation of claim 1, wherein In step (5), the specific process of back-projection of the instance segmentation result combined with the monocular depth estimation value and the camera intrinsic matrix is as follows: (4.1) the pixel points p after instance segmentation are converted into homogeneous coordinate form p=(u,v,1), wherein u and v are the horizontal and vertical coordinates of the pixel points in the two-dimensional image in pixel units; (4.2) Compute the monocular depth estimate d p = D t (p) and the camera intrinsic matrix k obtained in step (1), to obtain the coordinates (U, V, W, 1) of the pixel point p in the three-dimensional scene T = K -1 d p (u, v, 1) T , where U, V, W respectively correspond to the coordinate values on the x, y, z axes in the camera coordinate system, which are in meters, and finally obtain the regional three-dimensional point set of each instance.

5. The unsupervised monocular 3D object detection method based on video sequence and pre-trained instance segmentation of claim 1, wherein In step (6), the specific process of converting the region three-dimensional point set of each instance into the instance main three-dimensional point set and further calculating the three-dimensional bounding box is as follows: (5.1) first, the unsupervised clustering algorithm is applied to the region three-dimensional point set of each instance to obtain a plurality of clustering clusters; (5.2) the clustering cluster with the largest number of points is taken as the instance main three-dimensional point set, and the points contained in the other clustering clusters are removed as outliers; (5.3) based on the y-axis direction, the maximum and minimum values of the instance main three-dimensional point set are calculated, the difference between the two is taken as the y-axis height of the three-dimensional bounding box, and the minimum circumscribed rectangle of the instance main three-dimensional point set on the x-z plane is solved to finally obtain the three-dimensional bounding box.

6. The unsupervised monocular 3D object detection method based on video sequence and pre-trained instance segmentation of claim 2, wherein: For each current frame, the next frame of the current frame is selected, and for the self-supervised monocular depth estimation network, the sfm-learner, Monodepth series, sc-depth or packnet network structure is selected, wherein the network for pose estimation is modified to a ResNet-18 network with an input layer channel number of 6.

7. The unsupervised monocular three-dimensional object detection method based on a video sequence and pre-trained instance segmentation of claim 3, wherein: (7.1) the data set for pre-training of the instance segmentation network is selected as the MSCOCO data set, and the instance segmentation network uses a Mask R-CNN network structure; (7.2) for the result obtained after instance segmentation, a pixel region with a class confidence greater than or equal to 0.5 and a pixel confidence in the bounding box greater than or equal to 0.5 is selected to obtain the final instance segmentation result.

8. The unsupervised monocular three-dimensional object detection method based on a video sequence and pre-trained instance segmentation of claim 5, wherein: (8.1) for each instance region three-dimensional point set, a DBSCAN algorithm is used for unsupervised clustering, and in the algorithm parameters, the neighborhood radius E is set to 0.8 meters, and the minimum number of samples in the neighborhood MinPts is set to 10; (8.2) for the category that meets the assumption that the object is perpendicular to the ground, the instance main three-dimensional point set is used to solve the minimum circumscribed matrix in the x-z plane, and a rotation jamming algorithm is used. The computer readable storage medium comprises an unsupervised monocular three-dimensional object detection method based on a video sequence and pre-trained instance segmentation program, and when the processor executes the unsupervised monocular three-dimensional object detection method based on a video sequence and pre-trained instance segmentation program, the unsupervised monocular three-dimensional object detection method based on a video sequence and pre-trained instance segmentation of any one of claims 1 to 8 is realized. ​ ​ ​ 9. A non-transitory computer-readable storage medium, comprising: ​

Images