Object ellipsoid model single frame initialization method based on ellipse prediction model

Abstract

According to the object ellipsoid model single-frame initialization method based on the ellipse prediction model, ellipse detection frame information and Depth information are fully utilized, priori knowledge of a scene is not needed, single-frame initialization tasks of most objects can be completed, the unified orientation of an object coordinate system is not fixed, and follow-up optimization is facilitated; meanwhile, the RGB + Depth image is used as input, and the backbone network is modified into four convolutional layers, so that effective information input is increased, and the model complexity is reduced; a model loss function is modified into an improved normalized optimal transmission distance, and the evaluation effect on an ellipse prediction result is better.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision and deep learning, and particularly relates to a single-frame initialization method for an object ellipsoid model based on an ellipse prediction model. Background Art

[0002] In recent years, with the increasing improvement of traditional vision algorithms, as well as the improvement of the accuracy and the reduction of the cost of vision sensors, RGB cameras and depth cameras have been widely used in indoor scene positioning and mapping tasks, and algorithms such as ORB-SLAM2 have been able to achieve very good application effects in indoor and outdoor scenes with good lighting, stable camera changes, and rich scene textures. However, traditional vision positioning and mapping systems perform scene recognition and map construction based on feature point extraction algorithms. Both the positioning process and the constructed scene map only utilize limited scene information, making it difficult to perform higher-level application expansion. Moreover, feature point extraction and matching are greatly affected by factors such as lighting and perspective changes. Therefore, how to improve the feature extraction and representation of vision positioning and mapping has become one of the current research hotspots. To introduce high-dimensional features into the SLAM system, the most common method is to introduce semantic information, that is, object detection information or semantic segmentation information. Among them, since the training and application of object detectors have lower requirements for datasets and hardware computing power, the object detection results have become the main source of semantic information for current semantic SLAM systems.

[0003] Since the object detector can only obtain a rough detection box of a 2D object, the SLAM system introducing the object detector can only represent the object in a relatively rough manner. For example, the Cube-SLAM method uses a cube to represent the scene object, the QuadricSLAM and OA-SLAM use a dual quadratic surface (ellipsoid) to represent the scene object, and the SQ-SLAM uses a superquadric surface to represent the scene object. Among them, since the dual quadratic surface has a complete matrix representation and a simple geometric representation of the projection relationship, which is convenient for parameter optimization and projection matching, it has become the main research direction. The difficulties of the object-level SLAM system based on ellipsoid representation mainly lie in three aspects: the initialization of the ellipsoid model, parameter optimization, and data association. This patent mainly solves the first difficult problem. Different ellipsoid initialization methods use different semantic information inputs. Among them, the input based on the ellipse detection box is an emerging research hotspot. The following separately introduces the ellipse detection box generation method and the single-frame initialization method of the ellipsoid model:

[0004] 1) Ellipse Detection Box Generation

[0005] The traditional method for generating an elliptical detection box is to obtain an inscribed axis-aligned ellipse through the object detection box. For example, in the backend optimization stage of the OA-SLAM algorithm, this method is simple in design and can basically meet the usage requirements only in the backend optimization, but it obviously cannot meet the accuracy requirements in the model initialization. In addition, the paper "Object-Based Visual Camera Pose Estimation From Ellipsoidal Model and 3D-Aware Ellipse Prediction" proposed an ellipse prediction method based on CNN. By inputting the RGB image and the object detection result, the elliptical detection box of the object is predicted through the CNN model. The elliptical prediction box obtained by this method has higher accuracy than the previous method, but the model input does not contain Depth, and the Depth information can significantly distinguish foreground and background objects. Moreover, this algorithm is modified from the object detection model, with a huge number of model parameters and low model training efficiency.

[0006] 2) Ellipsoid parameter initialization

[0007] The ellipsoid model needs to maintain a total of 9 parameters. The traditional ellipsoid parameter initialization methods vary according to the input image information and semantic information, but all use the multi-frame initialization method. For monocular input and the input of the object detector detection result, geometric methods are used to calculate the ellipsoid parameters through no less than three observation frames with obvious perspective changes, or the sparse point cloud of the object is accumulated through multiple frames, and then the outliers are removed from the object-level semi-dense point cloud, the direction is calculated through PAC analysis, and the center coordinates and scale values are calculated through the edge points, such as OA-SLAM. For RGB-D input and the input of the instance segmentation model detection result, the ellipse fitting is generally realized by extracting the contour points from the instance segmentation masks of multiple-frame observations, such as QISO-SLAM. In addition, in order to simplify the computational complexity, many algorithms manually introduce the horizontal normal vector of the scene and stipulate that the ellipsoid models are all vertically placed on the horizontal plane to improve the accuracy of model construction.

[0008] It can be seen that in terms of obtaining the elliptical detection box, traditional methods mostly use the object detection box as the object detection result for constructing the ellipsoid model, and it is difficult to accurately determine the matching relationship between the projected ellipse of the ellipsoid and the rectangular detection box. Moreover, directly fitting the ellipse based on the rectangular detection box cannot achieve the accurate fitting of the ellipse to the object in the image, resulting in low accuracy in constructing the object-level ellipsoid model based on the rectangular detection box. In terms of ellipsoid parameter initialization, when the existing technology uses RGB-D and object detection as input, the multi-frame initialization method is mostly used. Such methods often fail to observe objects with fewer observations and limited observation angles, and the model construction speed of the ellipsoid initialization method based on multi-frame observations is slow, making it difficult to achieve real-time synchronous construction of the object-level map. Summary of the Invention

[0009] To solve the above problems, the present invention provides a single-frame initialization method for the ellipsoid model of an object based on an ellipse prediction model. By using the ellipse detection box information and Depth information, the single-frame initialization task of most objects can be achieved.

[0010] A single-frame initialization method for the ellipsoid model of an object based on an ellipse prediction model includes the following steps:

[0011] S1: Input the RGB image and Depth image of the current scene containing the target object into the ellipse prediction model to obtain the two-dimensional ellipse detection box of the target object;

[0012] S2: Perform segmentation based on K-means clustering on all Depth values within the two-dimensional ellipse detection box of the target object to distinguish the target object from the background;

[0013] S3: Determine whether to initialize the ellipsoid model for the target object included in the current scene according to whether the relative relationship between the Depth values of the target object and the background meets the set conditions.

[0014] Further, in step S2, performing segmentation based on K-means clustering on all Depth values within the two-dimensional ellipse detection box of the target object to distinguish the target object from the background specifically includes:

[0015] Randomly sample depth values uniformly within the two-dimensional ellipse detection box of the Depth image and store them in an array;

[0016] Remove the noise data with depth values less than 0.5m and greater than 5m from the array;

[0017] Use the K-means algorithm to cluster all the remaining valid depth data in the array, and determine the group with the larger average value among the two groups of valid depth data obtained by clustering as the background depth data, and set the other group as the object depth data;

[0018] Regard the regions corresponding to all the background depth data as the background, and regard the regions corresponding to all the object depth data as the target object.

[0019] Further, in step S3, the relative relationship between the Depth values of the target object and the background includes:

[0020] The mean value mean of the background depth data bg and the mean value mean of the object depth data obj and the proportion r of the number of removed noise data to the total amount of all depth data within the two-dimensional ellipse detection boxbad , the ratio r of the background depth data to all valid depth data within the two-dimensional elliptical detection frame bg .

[0021] Further, in step S2, it is determined whether to initialize the ellipsoid model for the target object included in the current scene according to whether the relative relationship between the Depth values of the target object and the background satisfies the set conditions, specifically as follows:

[0022] Judgment is made according to the ratio r bad , where, if r bad > 0.4, it is considered that the amount of valid depth data in the Depth image of the current scene is too small, so the ellipsoid model is not initialized for the current scene;

[0023] Judgment is made according to the ratio r bg , where, if r bg > 0.9, it is considered that the object depth data included in the Depth image of the current scene corresponds to an occluded object, and the background depth data is the real object depth data, so let mean obj = mean bg and then initialize the ellipsoid model for the current scene;

[0024] Judgment is made according to the mean mean bg , the mean mean obj , where, if meano bg - mean obj > thresh1, and r bg > 0.5, it is considered that the proportion of the background depth data is too large, so the ellipsoid model is not initialized for the current scene, and thresh1 is a set threshold; if mean bg - mean obj < thresh2, it is considered that there is no obvious background, and all valid depth data are used as object depth data, and mean obj is recalculated and then the ellipsoid model is initialized for the current scene, and thresh2 is a set threshold;

[0025] If none of the above is satisfied, the currently calculated mean obj is used to initialize the ellipsoid model for the current scene.

[0026] Further, in step S3, the method for initializing the ellipsoid model for the target object included in the current scene is:

[0027] S31: Calculate the coordinates (c xc , c yc ) of the center of the three-dimensional ellipsoid model in the normalized imaging plane:

[0028] c xc = (c x - d x ) / f x

[0029] c yc = (c y - d y ) / f y

[0030] where d x and d y represent the offsets of the principal point of the image plane in the x - direction and y - direction of the image coordinate system, respectively, in pixels, and f x and f y represent the equivalent focal lengths of the camera in the x - direction and y - direction of the image coordinate system, respectively, in pixels;

[0031] S32: Calculate the center coordinates C xc , c yc ) of the three - dimensional ellipsoid model: co (C x , C y , C z ):

[0032] C x = c xc * mean obj

[0033] C y = c yc * mean obj

[0034] C z = mean obj

[0035] S33: Obtain the rotation matrix R co = [x c , y c , z c from the ellipsoid coordinate system of the three - dimensional ellipsoid model to the camera coordinate system:

[0036]

[0037] y c = z c × x c

[0038] S34: Obtain the semi - axis lengths α, β, and γ of the three - dimensional ellipsoid model in the XYZ directions:

[0039] α = (x max - xmin ) / f x *mean obj *0.5

[0040] β = (y max - y min ) / f y *mean obj *0.5

[0041] γ = diff * 0.5

[0042] Wherein, x min and x max are respectively the minimum value and the maximum value in the X-axis direction of the two-dimensional ellipse detection frame, y min and y max are respectively the minimum value and the maximum value in the Y-axis direction of the two-dimensional ellipse detection frame, and diff is the difference between the maximum value and the minimum value among all valid depth data.

[0043] Furthermore, in step S1, the loss function used when training the ellipse prediction model is:

[0044]

[0045] Wherein, N1(μ1, Σ1) is the two-dimensional normal distribution that the predicted value of the two-dimensional ellipse detection frame output by the ellipse prediction model conforms to, μ1 is the center coordinate of the predicted value of the two-dimensional ellipse detection frame, and Σ1 is the variance of the two-dimensional normal distribution N1(μ1, Σ1), and there is θ1 is the angle between the major axis of the predicted value of the two-dimensional ellipse detection frame and the positive direction of the x-axis of the image coordinate system, (α1, β1) are the semi-axis lengths of the predicted value of the two-dimensional ellipse detection frame; N2(μ2, Σ2) is the two-dimensional normal distribution that the true value of the two-dimensional ellipse detection frame conforms to, μ2 is the center coordinate of the true value of the two-dimensional ellipse detection frame, and Σ2 is the variance of the two-dimensional normal distribution N2(μ2, Σ2), and there is θ2 is the angle between the major axis of the true value of the two-dimensional ellipse detection frame and the positive direction of the x-axis of the image coordinate system, (α2, β2) are the semi-axis lengths of the true value of the two-dimensional ellipse detection frame; represents the square of the L2 norm, Tr(·) represents the trace of the matrix, and Z is a set hyperparameter.

[0046] Furthermore, the method for obtaining the true value of the two-dimensional ellipse detection frame is:

[0047] The three-dimensional ellipsoid model of the target object is projected into each observation frame of the target object by using the camera projection model, and the obtained two-dimensional projection is the true value of the two-dimensional ellipse detection frame.

[0048] Further, the method for obtaining the camera projection model is as follows:

[0049] For each target object O i , multiple observation frames are selected For each observation frame F f , f = 1, 2, …, n, where n is the number of frames of the observation frames corresponding to the target object O i , the detection box of the target object O i is marked and the initial elliptical detection box C i of the target object O i is obtained by means of an axis-aligned ellipse inscribed in the detection box;

[0050] The following camera projection model is constructed:

[0051]

[0052] where P f is the pose transformation matrix of the target object in the f-th observation frame relative to the camera, T represents the transpose, C if is the initial elliptical detection box of the target object O i in the observation frame F f , is the dual representation of the initial elliptical detection box C i of the target object O f in the observation frame F if , β if is the proportionality coefficient corresponding to the target object O f in the observation frame F i , Q i is the three-dimensional ellipsoid model parameter to be solved for the target object O i , is the dual representation of the three-dimensional ellipsoid model parameter Q i ; among them, the three-dimensional ellipsoid model parameter Q i includes the center coordinates C co (C x , C y , C z ) of the three-dimensional ellipsoid model, the rotation matrix R co = [x c , y c , z c of the three-dimensional ellipsoid model, and the semi-axes lengths α, β, and γ of the three-dimensional ellipsoid model in the XYZ directions;

[0053] Select at least 10 observation frames to construct a camera projection model equation system, and solve for the three-dimensional ellipsoid model parameters corresponding to the target object O i ;

[0054] The method for obtaining the true values of the two-dimensional elliptical detection boxes corresponding to the respective observation frames is as follows:

[0055] The target object O i The corresponding three-dimensional ellipsoid model parameters Q i of the target object O i and the pose transformation matrix P corresponding to the target object O in the f-th observation frame f and the scale factor β if are substituted into the camera projection model to inversely solve the true value C of the two-dimensional ellipse detection frame of the target object O i in the f-th observation frame if–real .

[0056] Furthermore, after obtaining the true values of the two-dimensional ellipse detection frames of the target object in each observation frame, the true values of the two-dimensional ellipse detection frames, the RGB images, and the Depth images corresponding to each observation frame are respectively transformed to the set size, and then the true values of the two-dimensional ellipse detection frames, the RGB images, and the Depth images of the set size are used for the training of the ellipse prediction model;

[0057] Among them, the perspective transformation formulas of the RGB image and the Depth image are as follows:

[0058] p if ′ = H f p if

[0059] where p if is the coordinate of the point before transformation on the RGB image or Depth image corresponding to the target object O i in the f-th observation frame, and p if ′ is the coordinate of the point after transformation on the RGB image or Depth image corresponding to the target object O i in the f-th observation frame; H f is the perspective transformation matrix;

[0060] The perspective transformation formula of the true value of the two-dimensional ellipse detection frame is as follows:

[0061] C if – real *′ = H f C if–real *H f T

[0062] where C if–real is the true value of the two-dimensional ellipse detection frame before transformation corresponding to the target object O i in the f-th observation frame, and C if–real ′ is the true value of the two-dimensional ellipse detection frame after transformation corresponding to the target object O i in the f-th observation frame.

[0063] Furthermore, the elliptical prediction model is composed of a backbone network, a fully connected layer, a first detection head, and a second detection head;

[0064] The backbone network is used to extract RGB features and Depth features of the RGB image and the Depth image of the current scene containing the target object;

[0065] The fully connected layer is used to perform feature fusion on the RGB features and the Depth features to obtain fused features;

[0066] The first detection head is used to obtain the angle between the major axis of the two-dimensional elliptical detection frame of the target object and the positive direction of the x-axis of the image coordinate system according to the fused features;

[0067] The second detection head is used to obtain the center coordinates and semi-axis lengths of the two-dimensional elliptical detection frame of the target object according to the fused features.

[0068] Beneficial effects:

[0069] 1. The present invention provides a single-frame initialization method for an object ellipsoid model based on an elliptical prediction model, which makes full use of elliptical detection frame information and Depth information, and does not require prior knowledge of the scene. It can complete the single-frame initialization task of most objects, and does not fix the unified orientation of the object coordinate system, which is convenient for subsequent optimization.

[0070] 2. The present invention provides a single-frame initialization method for an object ellipsoid model based on an elliptical prediction model, which uses RGB+Depth images as input, modifies the backbone network to a 4-layer convolutional layer, which not only increases the input of effective information but also reduces the model complexity; modifies the model loss function to an improved normalized optimal transport distance, which has a better evaluation effect on the elliptical prediction results. Description of the drawings

[0071] Figure 1 is a flowchart of a single-frame initialization method for an object ellipsoid model based on an elliptical prediction model provided by the present invention;

[0072] Figure 2 is a structural diagram of the elliptical prediction model provided by the present invention;

[0073] Figure 3 is the prediction result of the elliptical prediction model provided by the present invention;

[0074] Figure 4 is an object-level map obtained by using the single-frame initialization method of the object model proposed by the present invention. Detailed implementation manners

[0075] To enable those skilled in the art to better understand the solution of this application, the technical solution in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application.

[0076] The present invention provides a single-frame initialization method for an object ellipsoid model based on an ellipse prediction model. The method flow is as Figure 1 , and its main process is mainly divided into two parts: the first part is as Figure 1 shown in the left process, which is the part of building and training the ellipse prediction model, mainly to complete the generation of the ellipse detection box of the object in the 2D image. The input of this model is the RGB image, the Depth image, and the object detection result of the RGB image. The generation area of the ellipse detection box is restricted by the detection result, and the depth and color information are provided by the Depth image and the RGB image, so that the model can generate a high-precision ellipse bounding box with a relatively small number of parameters. The second part is as Figure 1 shown in the right process, which is to realize the efficient and accurate initialization of the ellipsoid parameters based on the RGB image, the Depth image, and the ellipse detection box obtained by the ellipse prediction model. By calculating the dense point cloud of the current frame, extracting the point cloud in the ellipse detection box and performing clustering classification based on the depth value, removing noise points and background points, and then realizing single-frame ellipsoid initialization through the object dense point cloud. The detailed steps are as follows:

[0077] S1: Input the RGB image and the Depth image of the current scene containing the target object into the ellipse prediction model to obtain the two-dimensional ellipse detection box of the target object;

[0078] S2: Perform segmentation based on K-means clustering on all Depth values within the range of the two-dimensional ellipse detection box of the target object to distinguish the target object and the background, specifically as follows:

[0079] Randomly sample depth values uniformly within the two-dimensional ellipse detection box of the Depth image and store them in an array;

[0080] Remove the noise data with depth values less than 0.5m and greater than 5m in the array;

[0081] Use the K-means algorithm to cluster all the remaining valid depth data in the array, and determine the group with the larger average value in the two groups of valid depth data obtained by clustering as the background depth data, and set the other group as the object depth data;

[0082] Regard the areas corresponding to all the background depth data as the background, and regard the areas corresponding to all the object depth data as the target object.

[0083] S3: Determine whether to perform ellipsoid model initialization on the target object included in the current scene based on whether the relative relationship between the Depth values of the target object and the background meets the set conditions.

[0084] It should be noted that the relative relationship between the Depth values of the target object and the background includes:

[0085] The mean of the background depth data bg , the mean of the object depth data obj , the proportion r of the number of removed noise data to all depth data within the two-dimensional ellipse detection frame bad , the proportion r of the background depth data to all valid depth data within the two-dimensional ellipse detection frame bg .

[0086] Furthermore, determining whether to perform ellipsoid model initialization on the target object included in the current scene based on whether the relative relationship between the Depth values of the target object and the background meets the set conditions is as follows:

[0087] Judge according to the proportion r bad , where, if r bad > 0.4, it is considered that the amount of valid depth data in the Depth image of the current scene is too small, so the current scene does not perform ellipsoid model initialization;

[0088] Judge according to the proportion r bg , where, if r bg > 0.9, it is considered that the object depth data included in the Depth image of the current scene corresponds to an occluded object, and the background depth data is the real object depth data, so let mean obj = mean bg and then perform ellipsoid model initialization on the current scene;

[0089] Judge according to the mean mean bg , the mean mean obj , where, if mean bg - mean obj > thresh1, and r bg > 0.5, it is considered that the proportion of background depth data is too large and it is not a good initialization observation, so the current scene does not perform ellipsoid model initialization, and thresh1 is a set threshold; if mean bg - mean obj < thresh2, it is considered that there is no obvious background, and all valid depth data are used as object depth data, and the mean objThen, initialize the ellipsoid model for the current scene, where thresh2 is the set threshold;

[0090] If none of the above is satisfied, then use the currently calculated mean obj Initialize the ellipsoid model for the current scene.

[0091] Furthermore, obtain the minimum value x of the ellipse boundary in the X-axis direction min and the maximum value x max , the minimum value y in the Y-axis direction min and the maximum value y max , and the ellipse center coordinates (c x , c y ). Given the known camera internal parameters, the ellipse initialization parameters can be obtained; based on this, the method for initializing the ellipsoid model for the target object included in the current scene specifically includes the following steps:

[0092] S31: Calculate the coordinates (c xc , c yc ) of the center of the three-dimensional ellipsoid model in the normalized imaging plane:

[0093] c xc = (c x - d x ) / f x

[0094] c yc = (c y - d y ) / f y

[0095] where d x and d y respectively represent the offsets of the principal point of the image plane in the x-direction and y-direction of the image coordinate system, with the unit of pixel, and f x and f y respectively represent the equivalent focal lengths of the camera in the x-direction and y-direction of the image coordinate system, with the unit of pixel;

[0096] S32: Calculate the center coordinates C xc , c yc ) of the three-dimensional ellipsoid model according to the coordinates (c co (C x , C y , C z ):

[0097] C x = c xc * mean obj

[0098] C y = cyc *mean obj

[0099] C z = mean obj

[0100] S33: Since the present invention performs a certain screening on the initialization frame of the ellipsoid, it can be considered that the viewing angle of the frame used for object initialization is approximately parallel to the direction of the main view of the object. That is, the present invention can consider that the z-axis of the object-level coordinate system of the object initialization ellipsoid model coincides with the line connecting the camera optical center and the center of the ellipsoid. In addition, in order to simplify the calculation of the scale value of the ellipsoid, the present invention assumes that there is no roll angle and yaw angle between the ellipsoid coordinate system and the camera coordinate system. That is, the X-axis of the initialized ellipsoid model is parallel to the XOZ plane of the camera coordinate system. Therefore, after obtaining the direction vector of the Z-axis of the ellipsoid object coordinate system, the direction vector of the Y-axis can be obtained. After determining the directions of the Z-axis and the Y-axis, the direction vector of the X-axis can be obtained.

[0101] The rotation matrix R from the ellipsoid coordinate system to the camera coordinate system of the three-dimensional ellipsoid model co = [x c , y c , z c . The specific mathematical expression is as follows:

[0102]

[0103] y c = z c × x c

[0104] The above formula calculates the three direction vectors of the ellipsoid coordinate system, and combining them by columns together gives the rotation matrix R co = [x c , y c , z c .

[0105] S34: After obtaining the object-level coordinate system of the ellipsoid, the scale value in each axis direction can be calculated, that is, the semi-axis lengths of the initialized ellipsoid in the three axis directions are calculated. Since the present invention assumes that the Z-axis direction of the object-level coordinate system of the initialized ellipsoid is the direction of the light ray passing through the center of the ellipsoid, and there is no roll angle and yaw angle between the object-level coordinate system of the ellipsoid and the camera coordinate system, the scales in the X and Y directions of the ellipsoid can be directly obtained through the scale projections in the X and Y directions of the elliptical detection frame, and the scale in the Z direction of the ellipsoid is the "thickness" of the ellipsoid, that is, diff. The semi-axis lengths α, β, and γ in the XYZ directions of the three-dimensional ellipsoid model are specifically expressed mathematically as follows:

[0106] α = (x max - x min ) / fx *mean obj *0.5

[0107] β = (y max - y min ) / f y *mean obj *0.5

[0108] γ = diff * 0.5

[0109] Wherein, x min and x max are respectively the minimum value and the maximum value in the X-axis direction of the two-dimensional ellipse detection frame, y min and y max are respectively the minimum value and the maximum value in the Y-axis direction of the two-dimensional ellipse detection frame, diff is the difference between the maximum value and the minimum value in all valid depth data, representing the value range of the depth value of the object, that is, the "thickness" of the object under the observation angle of the current observation frame.

[0110] So far, the ellipse detection frame obtained by the present invention based on the ellipse prediction model realizes the single-frame initialization of the ellipsoid model of the object.

[0111] The following details the training process of the ellipse prediction model of the present invention.

[0112] Step 1: Preparation of the training data set.

[0113] The three-dimensional ellipsoid model of the target object is projected into each observation frame of the target object by using the camera projection model, and the obtained two-dimensional projection is the true value of the two-dimensional ellipse detection frame.

[0114] The method for obtaining the camera projection model is as follows:

[0115] For each target object O i , multiple observation frames are selected For each observation frame F f , f = 1, 2,..., n, n is the number of frames of the observation frames corresponding to the target object O i , the detection frame of the target object O i is marked and the initial ellipse detection frame C i of the target object O i is obtained by the method of aligning the inner tangent axis of the detection frame with an ellipse;

[0116] The following camera projection model is constructed:

[0117]

[0118] Wherein, P f is the pose transformation matrix of the target object in the f-th observation frame relative to the camera, T represents the transpose, Cif is the initial elliptical detection box of the target object O i in the observation frame F f . is the initial elliptical detection box C of the target object O i in the observation frame F f . The dual representation of C is β if where β if is the observation frame F f in which the target object O i corresponds to the scale factor Q i is the three-dimensional ellipsoid model parameter corresponding to the target object O to be solved i . is the three-dimensional ellipsoid model parameter Q i whose dual representation is; among them, the three-dimensional ellipsoid model parameter Q i includes the center coordinates C of the three-dimensional ellipsoid model co (C x , C y , C z ), the rotation matrix R of the three-dimensional ellipsoid model co = [x c , y c , z c , and the semi-axes lengths α, β, and γ of the three-dimensional ellipsoid model in the XYZ directions; that is to say, the degree of freedom of the three-dimensional ellipsoid model parameter Q i is 9, which are the 9 parameters that need to be initialized in a single frame. Therefore, at least 3 observation frames are required to solve the ellipsoid parameters; to ensure the construction accuracy of the ellipsoid model of the scene object, 10 observation frames are taken to construct an overdetermined equation for solution to obtain the ellipsoid model of the scene object

[0119] Based on this, at least 10 frames of observation frames are selected to construct a camera projection model equation set, and the three-dimensional ellipsoid model parameters corresponding to the target object O i are solved, as shown in Figure 4 .

[0120] After that, the above camera projection model can be used to project the object ellipsoid model into each observation frame to obtain a true value of the 2D elliptical detection box of the object, specifically

[0121] Substitute the three-dimensional ellipsoid model parameter Q corresponding to the target object O i , the pose transformation matrix P corresponding to the target object O i in the f-th observation frame, and the scale factor β i into the camera projection model f and solve for the true value C of the 2D elliptical detection box of the target object O if in the f-th observation frame . i if–real .​

[0122] Step 2, Perspective transformation.

[0123] It should be noted that for each type of object to be detected, the RGB image and Depth image of the current scene are cropped and distorted according to a specific size, so that the true value of the two-dimensional ellipse detection box, the RGB image, and the Depth image are distorted into images of size 256*256. Since the true value of the ellipse detection box of the corresponding object also needs to be distorted in the same proportion after the image is distorted, in order to facilitate calculation, the perspective transformation is used to represent the above distortion process. Let the 3*3 perspective transformation matrix of the f-th frame be H f , then the perspective transformation formulas for the RGB image and the Depth image are as follows:

[0124] p if ′ = H f p if

[0125] where H f is the perspective transformation matrix; p if is the coordinate of the point before transformation of the target object O i on the RGB image or Depth image corresponding to the f-th frame observation frame, and p if ′ is the coordinate of the point after transformation of the target object O i on the RGB image or Depth image corresponding to the f-th frame observation frame; homogeneous coordinates are used. Given that the degree of freedom of the perspective transformation is 8, at least four groups of points are required to calculate the perspective transformation matrix. Therefore, the four corner points of the input detection box are used as the origin of the perspective transformation, and the four corner points of the 256*256 image are used as the end points of the perspective transformation, and the perspective transformation matrix H f can be calculated. In engineering implementation, the perspective transformation matrix can be obtained through the getPerspectiveTransform() function of the OpenCV library.

[0126] After the perspective transformation of the RGB image or Depth image, the true value of the two-dimensional ellipse detection box also needs to be perspectively transformed. The perspective transformation formula of the true value of the two-dimensional ellipse detection box is as follows:

[0127] C if–real *′ = H f C if–real * H f T

[0128] where C if–real is the true value of the two-dimensional ellipse detection box before transformation of the target object O i corresponding to the f-th frame observation frame, and C if–real ′ is the target object Oi The true value of the transformed two-dimensional ellipse detection box corresponding to the f-th frame observation frame.

[0129] In addition, when calculating the perspective transformation matrix, the transformation origin can be filled with a certain boundary based on the corner points of the bounding box. For different objects to be detected, the optimal filled pixels are not the same. However, by putting the pixels around the detection box into the model for training, the accuracy of ellipse prediction can be improved to a certain extent because some ellipse detection results are not completely within the rectangular detection box.

[0130] Step 3: Construct the ellipse prediction model.

[0131] The ellipse prediction model is as Figure 2 shown. The input of the ellipse prediction model is a tensor of 4*256*256. The model consists of a backbone network for feature extraction, a fully connected layer for feature fusion, an output head for predicting the ellipse center and scale respectively, and an output head for predicting the ellipse angle. Since the input already contains the detection box information, the backbone network is designed very simply, only including 4 convolutional layers. After the first three convolutional layers, a max-pooling layer is connected to reduce the size of the feature map. After the last convolutional layer, an adaptive average pooling layer is connected to adjust the size of the feature map to (2*2). The input of the fully connected layer is obtained by unfolding the output feature map from the convolutional layer. After unfolding, the feature dimension is 2028. The fully connected layer includes 3 linear layers, and each linear layer is followed by a batch normalization layer to achieve regularization, avoid gradient disappearance or explosion, and accelerate the training convergence. Among the five prediction parameters of the ellipse, since the range of the angle parameter is different from the other four parameters, a separate output head is set for the angle parameter, while the other four parameters share a prediction head. The output head also adopts the form of an MLP layer. For the output heads of the ellipse center and scale values, in the final output, a Sigmoid layer is used to normalize the output to 0-1, that is, if the obtained prediction value is (c x ,c y ,α,β), then the true value in the 256*256 image should be (c x *256,c y *256,α*256,β*256). For angle prediction, since in the present invention, the angle θ is set as the included angle between the major axis of the ellipse and the positive direction of the x-axis, and the value range is Therefore, by taking the Tanh of the output of the MLP layer of the angle output head, the output can be restricted to (-1,1), and then the radian value of the predicted angle can be obtained through the arcsine function.

[0132] The loss function of the ellipse prediction model adopts a distance function similar to the matching accuracy evaluation between ellipses in object-level SLAM, that is, the improved normalized optimal transport distance. For any ellipse, if the center coordinates of the ellipse (c x , c y ), the semi-axis lengths (α, β), and the angle θ between the major axis of the ellipse and the positive x-axis direction of the image coordinate system are known, then the ellipse can be represented as a two-dimensional normal distribution N(μ, Σ), where the definitions of μ and Σ are as follows:

[0133]

[0134] Therefore, for the predicted value of the two-dimensional ellipse detection box and the true value of the two-dimensional ellipse detection box, its improved normalized optimal transport distance loss function is as follows:

[0135]

[0136] Among them, N1(μ1, Σ1) is the two-dimensional normal distribution that the predicted value of the two-dimensional ellipse detection box output by the ellipse prediction model conforms to, μ1 is the center coordinate of the predicted value of the two-dimensional ellipse detection box, and Σ1 is the variance of the two-dimensional normal distribution N1(μ1, Σ1), and there is θ1 is the angle between the major axis of the predicted value of the two-dimensional ellipse detection box and the positive x-axis direction of the image coordinate system, (α1, β1) is the semi-axis length of the predicted value of the two-dimensional ellipse detection box; N2(μ2, Σ2) is the two-dimensional normal distribution that the true value of the two-dimensional ellipse detection box conforms to, μ2 is the center coordinate of the true value of the two-dimensional ellipse detection box, and Σ2 is the variance of the two-dimensional normal distribution N2(μ2, Σ2), and there is θ2 is the angle between the major axis of the true value of the two-dimensional ellipse detection box and the positive x-axis direction of the image coordinate system, (α2, β2) is the semi-axis length of the true value of the two-dimensional ellipse detection box; represents the square of the L2 norm, Tr(·) represents the trace of the matrix, and Z is the set hyperparameter.

[0137] As Figure 3 shown, it is the prediction result of the ellipse prediction model. Among them, the blue ellipse is the true value of the two-dimensional ellipse detection box, and the green ellipse is the predicted value of the two-dimensional ellipse detection box, and the two basically completely overlap.

[0138] Step 4. After the ellipse prediction model is constructed, the batch size group of 256*256 RGB and Depth images can be stacked into a Tensor data of batch size*4*256*256 dimensions and input into the CNN model for model training. The initial learning rate for training is 5*10 -5 , and a dynamic learning rate adjustment strategy is adopted.

[0139] Step 5. After the training is completed, for the dataset to be mapped, first use the object detector to detect the rectangular frames of the target objects, and then input the data according to the input requirements of the ellipse prediction model to obtain the ellipse prediction results. For the ellipse prediction results, post-processing is required, which is equivalent to the inverse operation of the input processing, mainly involving the inverse normalization and inverse perspective transformation of the output ellipse parameters. After completing the above operations, the predicted ellipse prediction results can be stored in the input format of object-level SLAM.

[0140] It should be noted that the present invention is based on the ORB-SLAM2 framework. The single-frame initialization of the object ellipsoid model requires the tracking thread of ORB-SLAM2 to provide accurate camera poses. Therefore, the subsequent process only involves initializing the parameters of the ellipsoid in the camera coordinate system of the current frame.

[0141] In summary, the single-frame initialization method of the object ellipsoid model based on the ellipse prediction model provided by the present invention has the following advantages:

[0142] 1. In the field of single-frame initialization of the ellipse prediction model, most of the previous ellipsoid initialization methods use multi-frame initialization methods. Such methods can complete model initialization based on multiple observation frames with obvious perspective differences, and cannot complete model initialization in scenes with limited perspectives. In addition, a few single-frame initialization methods require the direction of the horizontal plane normal vector of the given world coordinate system, which is difficult for unknown scenes or real-time applications and requires other sensors to provide the direction of the horizontal plane normal vector. In comparison, the single-frame initialization method of the ellipse prediction model proposed by the present invention makes full use of the ellipse detection frame information and Depth information, and does not require prior knowledge of the scene. It can complete the single-frame initialization tasks of most objects, and does not fix the unified orientation of the object coordinate system, which is convenient for subsequent optimization.

[0143] 2. In the field of the ellipse prediction model technology, the previous ellipse prediction models use VGG19 as the backbone network structure, resulting in a very large number of model parameters. The model input only contains RGB information and does not utilize the useful information in the Depth image. The output loss function is the sampling clustering loss, and the evaluation effect on the ellipse prediction results is average. In comparison, the ellipse prediction model proposed by the present invention uses RGB+Depth images as the input, modifies the backbone network to a 4-layer convolutional layer, which not only increases the input of effective information but also reduces the model complexity; modifies the model loss function to the improved normalized optimal transport distance, and has a better evaluation effect on the ellipse prediction results.

[0144] Of course, the present invention can also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can certainly make various corresponding changes and deformations according to the present invention, but these corresponding changes and deformations should all fall within the protection scope of the appended claims of the present invention.

Claims

1. A single-frame initialization method for an object ellipsoid model based on an ellipse prediction model, characterized in that: The following steps are involved: S1: Input the RGB image and Depth image of the current scene containing the target object into the ellipse prediction model to obtain a two-dimensional ellipse detection box of the target object; S2: Segment all Depth values ​​within the two-dimensional elliptical detection frame of the target object based on K-means clustering to distinguish the target object from the background; S3: Determine whether to initialize the ellipsoid model of the target object included in the current scene according to whether the relative relationship between the depth values ​​of the target object and the background meets the set conditions.

2. A single-frame initialization method for an object ellipsoid model based on an ellipse prediction model as claimed in claim 1, characterized in that: In step S2, all Depth values ​​within the two-dimensional ellipse detection frame of the target object are segmented based on K-means clustering to distinguish the target object from the background. Specifically: Uniformly randomly sample depth values ​​within the two-dimensional ellipse detection box of the Depth image and store them in an array; Eliminate noise data with depth values ​​less than 0.5m and greater than 5m in the array; Use the K-means algorithm to cluster all the remaining valid depth data in the array. The two groups of depth valid data obtained by clustering with a larger average value are determined as background depth data, and the other group is set as object depth data; The area corresponding to all background depth data is used as the background, and the area corresponding to all object depth data is used as the target object.

3. A single-frame initialization method for an object ellipsoid model based on an ellipse prediction model as claimed in claim 2, characterized in that: In step S3, the relative relationship between the depth values ​​of the target object and the background includes: The mean of the background depth data bg 、The mean of the object depth data obj , the ratio of the number of removed noise data to the total depth data in the two-dimensional ellipse detection frame r bad , the ratio of background depth data to all valid depth data in the two-dimensional ellipse detection frame r bg .

4. A single-frame initialization method for an object ellipsoid model based on an ellipse prediction model as claimed in claim 3, characterized in that: In step S2, whether to initialize the ellipsoid model of the target object contained in the current scene is determined according to whether the relative relationship between the depth values ​​of the target object and the background meets the set conditions. Specifically, the following is the procedure: According to the ratio r bad Make a judgment, where if r bad >0.4, it is considered that the effective depth data of the Depth image of the current scene is too small, and the ellipsoid model will not be initialized for the current scene; According to the ratio r bg Make a judgment, where if r bg >0.9, it is considered that the object depth data contained in the depth image of the current scene corresponds to the occluded object, and the background depth data is the real object depth data, so let mean obj =mean bg Then initialize the ellipsoid model of the current scene; According to the mean bg and the mean obj to make a judgment. Among them, if mean bg - mean obj > thresh1 and r bg > 0.5, it is considered that the proportion of background depth data is too large, and the ellipsoid model initialization is not performed for the current scene. Thresh1 is a set threshold; if mean bg - mean obj < thresh2, it is considered that there is no obvious background, and all valid depth data are used as object depth data, and mean obj is recalculated and then the ellipsoid model initialization is performed for the current scene. Thresh2 is a set threshold; If none of the above is satisfied, the currently calculated mean is used. obj Initialize the ellipsoid model for the current scene.

5. A single-frame initialization method for an object ellipsoid model based on an ellipse prediction model as claimed in claim 4, characterized in that: In step S3, the method for initializing the ellipsoid model of the target object contained in the current scene is: S31: Calculate the coordinates of the center of the three-dimensional ellipsoid model in the normalized imaging plane (c xc ,c yc ): c xc =(c x -d x ) / f x c yc =(c y -d y ) / f y Among them, d x and d y Respectively represent the offset of the principal point of the image plane in the x-direction and y-direction of the image coordinate system, in pixels, f x and f y Respectively represent the equivalent focal length of the camera in the x-direction and y-direction of the image coordinate system, in pixels; S32: According to the coordinates (c xc ,c yc ) Calculate the center coordinates C of the three-dimensional ellipsoid model co (C x ,C y ,C z ): C x =c x c*mean obj C y =c yc *mean obj C z =mean obj S33: Get the rotation matrix R between the ellipsoid coordinate system of the three-dimensional ellipsoid model and the camera coordinate system co =[x c ,y c ,z c ]: y c =z c ×x c S34: Obtain the semi-axis lengths α, β and γ of the three-dimensional ellipsoid model in the XYZ directions: α=(x max -x min ) / f x *mean obj *0.5 β=(y max -y min ) / f y *mean obj *0.5 γ=diff*0.5 Among them, x min and x max are the minimum and maximum values ​​of the X-axis direction of the two-dimensional ellipse detection frame, and y min and max They are the minimum and maximum values ​​of the two-dimensional ellipse detection box in the Y-axis direction, and diff is the difference between the maximum and minimum values ​​of all valid depth data.

6. A single-frame initialization method for an object ellipsoid model based on an ellipse prediction model as claimed in claim 1, characterized in that: In step S1, the loss function used when training the ellipse prediction model is for: Among them, N1(μ1, Σ1) is the two-dimensional normal distribution of the two-dimensional ellipse detection box prediction value output by the ellipse prediction model, μ1 is the center coordinate of the two-dimensional ellipse detection box prediction value, and Σ1 is the variance of the two-dimensional normal distribution N1(μ1,Σ1), and θ1 is the angle between the major axis of the predicted value of the two-dimensional ellipse detection frame and the positive direction of the x-axis of the image coordinate system, (α1, β1) is the semi-axis length of the predicted value of the two-dimensional ellipse detection frame; N2(μ2, Σ2) is the two-dimensional normal distribution that the true value of the two-dimensional ellipse detection frame conforms to, μ2 is the central coordinate of the true value of the two-dimensional ellipse detection frame, and Σ2 is the variance of the two-dimensional normal distribution N2(μ2,Σ2), and θ2 is the angle between the major axis of the true value of the two-dimensional ellipse detection frame and the positive direction of the x-axis of the image coordinate system, (α2, β2) is the semi-axis length of the true value of the two-dimensional ellipse detection frame; represents the square of the L2 norm, Tr(·) represents the trace of the matrix, and Z is the set hyperparameter.

7. A single-frame initialization method for an object ellipsoid model based on an ellipse prediction model as claimed in claim 6, characterized in that: The method for obtaining the true value of the two-dimensional ellipse detection box is: The camera projection model is used to project the three-dimensional ellipsoid model of the target object into each observation frame of the target object, and the obtained two-dimensional projection is the true value of the two-dimensional ellipse detection frame.

8. A single-frame initialization method for an object ellipsoid model based on an ellipse prediction model as claimed in claim 6, characterized in that: The method to obtain the camera projection model is: For each target object O i , select multiple observation frames For each observation frame F f , f=1,2,…,n, n is the target object O i The corresponding observation frame number, marking the target object O i The detection frame is then used to obtain the target object O by aligning the ellipse with the tangent axis in the detection frame. i The initial ellipse detection box C i ; Construct the following camera projection model: Among them, P f is the pose transformation matrix of the target object in the f-th observation frame relative to the camera, T represents transposition, C if The target object O i In the observation frame F f The initial ellipse detection box in The target object O i In the observation frame F f The initial ellipse detection box C in if The dual representation of β if is the observation frame F f The target object O i The corresponding proportionality factor, Q i is the target object O to be solved i The corresponding three-dimensional ellipsoid model parameters, is the three-dimensional ellipsoid model parameter Q i The dual representation of ; where the three-dimensional ellipsoid model parameter Q i Including the center coordinates C of the three-dimensional ellipsoid model co (C x ,C y ,C z ), the rotation matrix R of the three-dimensional ellipsoid model co =[x c ,y c ,z c ], the semi-axis lengths α, β and γ of the three-dimensional ellipsoid model in the XYZ direction; Select at least 10 observation frames to construct the camera projection model equations and solve the target object O i Corresponding three-dimensional ellipsoid model parameters; The method for obtaining the true value of the two-dimensional ellipse detection frame corresponding to each observation frame is: The target object O i The corresponding three-dimensional ellipsoid model parameters Q i , target object O i The pose transformation matrix P corresponding to the observation frame at the fth frame f , proportionality coefficient β if Substitute the camera projection model Inversely solve the target object O i The true value C of the two-dimensional ellipse detection box in the f-th observation frame if–real .

9. A single-frame initialization method for an object ellipsoid model based on an ellipse prediction model as claimed in claim 7 or 8, characterized in that: After obtaining the true value of the two-dimensional ellipse detection frame of the target object in each observation frame, the true value of the two-dimensional ellipse detection frame, the RGB image and the Depth image corresponding to each observation frame are transformed to the set size respectively, and then the true value of the two-dimensional ellipse detection frame, the RGB image and the Depth image of the set size are used for the training of the ellipse prediction model; Among them, the perspective transformation formula of RGB image and Depth image is as follows: p if ′=H f p if Among them, p if The target object O i The coordinates of the point before transformation on the RGB image or Depth image corresponding to the observation frame of the fth frame, p if ′ is the target object O i The transformed point coordinates on the RGB image or Depth image corresponding to the f-th observation frame; H f is the perspective transformation matrix; The perspective transformation formula of the true value of the two-dimensional ellipse detection box is as follows: C if – real *′=H f C if–real *H f T Among them, C if–real The target object O i The true value of the two-dimensional ellipse detection box before the transformation corresponding to the f-th frame observation frame, C if–real ′ is the target object O i The true value of the transformed two-dimensional ellipse detection box corresponding to the observation frame at the f-th frame.

10. A single-frame initialization method for an object ellipsoid model based on an ellipse prediction model as claimed in claim 6, characterized in that: The ellipse prediction model consists of a backbone network, a fully connected layer, a first detection head and a second detection head; The backbone network is used to extract RGB features and Depth features of the RGB image and Depth image of the current scene containing the target object; The fully connected layer is used to fuse the RGB features and the Depth features to obtain fused features; The first detection head is used to obtain the angle between the major axis of the two-dimensional ellipse detection frame of the target object and the positive direction of the x-axis of the image coordinate system according to the fusion feature; The second detection head is used to obtain the center coordinates and semi-axis lengths of the two-dimensional ellipse detection frame of the target object according to the fusion features.

Images