Dynamic object pose estimation method based on dynamic contour key point tracking

Abstract

The invention provides a dynamic object pose estimation method based on dynamic contour key point tracking. The method comprises the following steps: extracting a dynamic object from video data by using an instance segmentation network and obtaining a mask of the dynamic object; generating an initial attitude of a dynamic object of an initial frame image in the current video clip based on a discrete sampling method; performing target tracking on a dynamic object of a non-initial frame image in the current video clip by using a dynamic contour key point tracking method, and performing pixel-level tracking on a dynamic contour key point by using a pixel tracking method; generating a preliminary pose estimation of a dynamic object of a frame image in the current video clip by using a PnP algorithm, and obtaining an estimation result of a camera self-motion pose in the current video clip by using an image positioning and mapping technology; and generating and optimizing the dynamic pose of the dynamic object in the current video by using the dynamic contour key point of the current video clip, the initial pose of the dynamic object, the pose preliminary estimation of the dynamic object and the estimation result of the self-motion pose of the camera.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a dynamic object posture estimation method based on dynamic contour key point tracking, an electronic device and a storage medium. Background Art

[0002] With the development of science and technology, robots are becoming more and more closely related to people's lives. Simultaneous localization and mapping (SLAM) technology is the core technology for robots to explore and navigate autonomously. Most of the existing SLAM technologies assume that the scene is absolutely static, which cannot meet the application requirements of real scenes. Therefore, both the industry and academia have an urgent need for 4D SLAM (3D+time), namely dynamic SLAM, especially in scene understanding and task decision-making. The current difficulty of dynamic (4D) SLAM is that it is impossible to establish long-term and stable spatiotemporal information associations for dynamic objects in the scene, which is of great significance for dynamic object tracking and the application of SLAM systems in real scenes. Therefore, the existing visual SLAM still has the following problems: it is impossible to establish long-term and robust spatiotemporal information associations for dynamic objects; the existing 6DoF pose estimation methods for dynamic objects have high computational complexity and cannot be well combined with feature point-based SLAM systems. Summary of the invention

[0003] In view of the above problems, the present invention provides a dynamic object pose estimation method based on dynamic contour key point tracking, an electronic device and a storage medium, in order to solve at least one of the above problems.

[0004] According to a first aspect of the present invention, a method for estimating a posture of a dynamic object based on dynamic contour key point tracking is provided, comprising:

[0005] Perform instance segmentation on each frame of the video data using an instance segmentation network to obtain an instance segmentation result for each frame of the image, extract dynamic objects from the instance segmentation result for each frame of the image, and filter the instance segmentation result for each frame of the image according to a predefined filtering condition to obtain a mask of the dynamic object for each frame of the image in the video data;

[0006] A fitting ground equation of an initial frame image in the current video clip is obtained by using a random sampling consensus algorithm, and an initial posture of a dynamic object in the initial frame image is generated according to the fitting ground equation of the initial frame image and a mask of the dynamic object in the initial frame image, wherein the initial frame image represents a frame image in which the dynamic object is first observed in the current video clip, and the current video clip represents continuous video frames intercepted from the video data according to predefined conditions;

[0007] Extracting semantic feature points on the contour of the dynamic object in the initial frame image by using a dynamic contour key point tracking method to obtain dynamic contour key points of the dynamic object in the initial frame image, and tracking the dynamic contour key points at the pixel level by using a pixel tracking method to obtain tracking results of dynamic contour key points of non-initial frame images in the current video clip;

[0008] Based on the tracking results of the dynamic contour key points of the non-initial frame image, the perspective-3D algorithm is used to generate a preliminary estimate of the pose of the dynamic object in the current video clip, and the image positioning and mapping technology is used to obtain the estimated result of the camera's self-motion pose in the current video clip;

[0009] The factor graph of the dynamic objects in the current video clip is constructed using the dynamic contour key points of the current video clip, the initial posture of the dynamic objects, the preliminary estimation of the posture of the dynamic objects and the estimation results of the camera's self-motion posture. The dynamic posture of the dynamic objects in the current video is generated by dynamically adjusting and optimizing the factor graph.

[0010] According to an embodiment of the present invention, the above-mentioned method for estimating the pose of a dynamic object based on dynamic contour key point tracking further includes:

[0011] The motion pose of the dynamic object in each video clip in the video data is optimized using a bundle adjustment optimization framework based on dynamic contour key point tracking to obtain the motion trajectory of the dynamic object in the video data.

[0012] According to an embodiment of the present invention, the instance segmentation network is used to perform instance segmentation on each frame of the video data to obtain an instance segmentation result of each frame of the image, and the dynamic object is extracted from the instance segmentation result of each frame of the image, and the instance segmentation result of each frame of the image is filtered according to a predefined filtering condition to obtain a mask of the dynamic object of each frame of the video data.

[0013] Acquire video data through a video acquisition device, wherein the video acquisition device includes a monocular camera, a binocular camera, and a depth camera;

[0014] Perform instance segmentation on each frame of the video data using an instance segmentation network, and extract dynamic objects from the instance segmentation results of each frame of the image to obtain the dynamic objects of each frame of the image, wherein the instance segmentation network includes a SOLO instance segmentation network;

[0015] Determine the predefined screening condition according to the overlap rate of the object masks between adjacent frame images in the video data and the ratio of the object pixels to the full image pixels in each frame image;

[0016] According to the predefined filtering conditions, static objects and objects smaller than a preset size are filtered out from the instance segmentation results of each frame of the image, and the mask of the dynamic objects in each frame of the image is obtained.

[0017] According to an embodiment of the present invention, the acquiring of video data by the video acquisition device includes:

[0018] The video data collected by the monocular camera is preprocessed through a monocular depth estimation network to obtain a depth map of each frame image in the video data.

[0019] According to an embodiment of the present invention, the above-mentioned obtaining the fitting ground equation of the initial frame image in the current video clip by using the random sampling consensus algorithm, and generating the initial posture of the dynamic object of the initial frame image according to the fitting ground equation of the initial frame image and the mask of the dynamic object of the initial frame image includes:

[0020] According to preset values, the pitch angle and roll angle of the moving object in the initial frame image are set, and the ground of the initial frame image is fitted using a random sampling consensus algorithm to obtain a fitting ground equation of the initial frame image;

[0021] According to the fitted ground equation of the initial frame image, the camera self-motion posture of the current video clip is adjusted, and the ground of the initial frame image is set based on the adjustment result and the world coordinate system;

[0022] The mask of the dynamic object in the initial frame image is projected into the world coordinate system, and the projection result is used to perform discrete sampling of the yaw angle of the dynamic object within a preset angle range on the two-dimensional pixel plane to obtain a discrete sampling result.

[0023] Calculate the minimum area bounding box of the discrete sampling result in the two-dimensional projection area, and use the minimum area bounding box of the discrete sampling result as the top view of the 3D bounding box of the dynamic object in the initial frame image to estimate the yaw angle, X-axis initial point, Y-axis initial point, length and width of the 3D bounding box of the dynamic object in the initial frame image;

[0024] Based on the constraint condition between the yaw angle of the dynamic object in the initial frame image and the 3D bounding box of the dynamic object in the initial frame image, the Z-axis initial point and height of the 3D bounding box of the dynamic object are calculated to obtain all parameter values ​​of the 3D bounding box of the dynamic object;

[0025] An initial posture of the dynamic object in the initial frame image is generated based on all parameter values ​​of the 3D bounding box of the dynamic object in the initial frame image, wherein the initial posture of the dynamic object is a posture with six degrees of freedom.

[0026] According to an embodiment of the present invention, the above-mentioned method of using dynamic contour key point tracking to extract semantic feature points on the contour of the dynamic object of the initial frame image, obtain the dynamic contour key points of the dynamic object of the initial frame image, and use the pixel tracking method to perform pixel-level tracking on the dynamic contour key points to obtain the tracking results of the dynamic contour key points of the non-initial frame image in the current video clip, including:

[0027] Extracting semantic feature points on the contour of the dynamic object in the initial frame image in the current video clip by using a dynamic contour key point tracking method to obtain dynamic contour key points of the dynamic object in the initial frame image;

[0028] Using the pixel tracking method to track each non-initial frame image at the pixel level, the matching tracking result of the dynamic contour key points of each non-initial frame image is obtained;

[0029] According to the preset key point screening conditions, the dynamic contour key points of the dynamic objects in each non-initial frame image are screened by using a random sampling consensus algorithm to obtain the screened dynamic contour key points of each non-initial frame image.

[0030] According to an embodiment of the present invention, the above-mentioned generating the dynamic pose of the dynamic object in the current video by dynamically adjusting and optimizing the factor graph includes:

[0031] Based on the tracking results of the dynamic contour key points of each non-initial frame image, detecting the change in the number of the dynamic contour key points of the current non-initial frame image, and obtaining the detection result;

[0032] When the detection result satisfies the predefined change condition, a new dynamic contour key point is obtained from the current non-initial frame image, and the current non-initial frame image is set as the map key frame image of the current video segment;

[0033] Adding a map keyframe image to the map of the current video segment and updating dynamic contour key points of the map keyframe image;

[0034] Based on the reprojection error of the map key frame image, the rigid body transformation of the dynamic objects in the map key frame image is parameterized using Lie algebra to obtain the reprojection error cost function;

[0035] The estimated result of the camera's self-motion pose of the current video clip is fixed, the weights in the factor graph are adjusted using a heuristic algorithm, and the preliminary estimate of the pose of the dynamic object in each non-initial frame image is optimized by minimizing the reprojection error cost function to generate the dynamic pose of the dynamic object in the current video.

[0036] According to a second aspect of the present invention, there is provided an electronic device, comprising:

[0037] one or more processors;

[0038] a storage device for storing one or more programs,

[0039] When one or more programs are executed by one or more processors, the one or more processors execute a dynamic object pose estimation method based on dynamic contour key point tracking.

[0040] According to a second aspect of the present invention, there is provided a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute a dynamic object pose estimation method based on dynamic contour key point tracking.

[0041] The above-mentioned dynamic object pose estimation method based on dynamic contour key point tracking provided by the present invention can obtain semantic information in video data in real time through an instance segmentation network, and can screen out dynamic objects in video data; the initial pose estimation of dynamic objects based on the discrete sampling method ensures the robust pose initialization of dynamic objects in the world coordinate system when they are observed for the first time. The pixel tracking method based on dynamic contour key points establishes a long-term robust spatiotemporal association of dynamic object contour points in a 2D pixel coordinate system. The three-dimensional-perspective algorithm based on dynamic contour key point tracking provides a preliminary dynamic object pose estimation, and the subsequent bundle adjustment optimization framework based on contour key point tracking combines the object contour key points, the three-dimensional coordinates of the spatial point, the camera pose, and the three-dimensional-perspective algorithm motion object pose estimation to optimize the dynamic object pose, and obtain accurate object motion, thereby generating an accurate dynamic object motion trajectory. The above-mentioned method provided by the present invention constructs a long-term robust spatiotemporal association of dynamic object contour key points and performs high-precision dynamic object pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of a method for estimating a posture of a dynamic object based on dynamic contour key point tracking according to an embodiment of the present invention;

[0043] Figure 2 is a flow chart of a method for dynamic object pose estimation and trajectory generation based on dynamic contour key point tracking according to an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of detection results of dynamic objects in video data using an instance segmentation network according to an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of a posture initialization process of a dynamic object based on discrete sampling according to an embodiment of the present invention;

[0046] Figure 5is a schematic diagram of a bundle adjustment optimization framework based on dynamic contour key point tracking according to an embodiment of the present invention;

[0047] Figure 6 A block diagram of an electronic device suitable for implementing a method for estimating a posture of a dynamic object based on dynamic contour key point tracking according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0049] In order to overcome the technical problems existing in the existing technical solutions, such as the defects of SLAM technology based on static scene assumption in real scene applications, the present invention provides a pose estimation method based on dynamic object contour key point tracking, which realizes the establishment of long-term robust spatiotemporal information association for dynamic objects in the scene and performs high-precision online trajectory estimation and generation; the scheme fully considers the real-time and high-precision requirements of robot operation in dynamic scenes, and designs a multi-dynamic target tracking and trajectory estimation method with strong portability, low cost and high accuracy; at the same time, the method also has the characteristics of wide applicability and strong portability.

[0050] Figure 1 The present invention is a flowchart of a method for estimating the posture of a dynamic object based on dynamic contour key point tracking according to an embodiment of the present invention.

[0051] like Figure 1 As shown, the above-mentioned method for estimating the pose of a dynamic object based on dynamic contour key point tracking includes operations S110 to S150.

[0052] In operation S110, an instance segmentation network is used to perform instance segmentation on each frame of the video data to obtain an instance segmentation result for each frame of the image, dynamic objects are extracted from the instance segmentation result of each frame of the image, and the instance segmentation result of each frame of the image is filtered according to predefined filtering conditions to obtain a mask of the dynamic object for each frame of the image in the video data.

[0053] The above predefined filtering conditions mean that static objects and small objects in the scene are filtered out based on instance segmentation according to the Intersection over Union (IoU) of objects between frames and the ratio of their pixels to the pixels of the entire image.

[0054] In operation S120, a fitting ground equation of the initial frame image in the current video clip is obtained using a random sampling consensus algorithm, and an initial posture of the dynamic object in the initial frame image is generated according to the fitting ground equation of the initial frame image and the mask of the dynamic object in the initial frame image.

[0055] The initial frame image represents a frame image in which a dynamic object is first observed in the current video segment, and the current video segment represents continuous video frames captured from video data according to predefined conditions.

[0056] The above predefined condition means that the frame image currently being processed is used as a node, and a plurality of continuous frame images before the current frame image are intercepted from the video data to obtain the current video segment.

[0057] For example, if the currently processed frame image is the 50th frame image of the video data, then the current video segment is obtained by intercepting a plurality of consecutive frame images (eg, 10 consecutive frame images) before the 50th frame image.

[0058] Among them, the initial posture of the dynamic object has six degrees of freedom (6DoF).

[0059] In operation S130, a dynamic contour key point tracking method is used to extract semantic feature points on the contour of the dynamic object in the initial frame image to obtain the dynamic contour key points of the dynamic object in the initial frame image, and a pixel tracking method is used to track the dynamic contour key points at the pixel level to obtain the tracking results of the dynamic contour key points of the non-initial frame image in the current video clip.

[0060] In operation S140, based on the tracking results of the dynamic contour key points of the non-initial frame image, a perspective-3D algorithm is used to generate a preliminary estimate of the pose of the dynamic object in the current video clip, and the image positioning and mapping technology is used to obtain the estimated result of the camera self-motion pose in the current video clip.

[0061] The above-mentioned image positioning and mapping technology (Simultaneous Localization and Mapping) is an algorithm for simultaneous visual positioning and map construction. It realizes camera positioning and map construction by extracting feature information from the image and using methods such as feature matching and posture calculation.

[0062] The above tracking results represent the corresponding relationship between the dynamic contour key points between two adjacent frames in the current video clip.

[0063] The above perspective-three-dimensional algorithm represents a PnP (Perspective-n-Point) algorithm, which is used to solve the correspondence between 3D points and 2D points. Those skilled in the art may adopt other algorithms for solving the correspondence between 3D points and 2D points according to actual needs.

[0064] In operation S150, a factor graph of the dynamic objects in the current video clip is constructed using the dynamic contour key points of the current video clip, the initial posture of the dynamic objects, the preliminary estimation of the posture of the dynamic objects, and the estimation results of the camera's self-motion posture, and the dynamic posture of the dynamic objects in the current video is generated by dynamically adjusting and optimizing the factor graph.

[0065] According to an embodiment of the present invention, the above-mentioned method for estimating the pose of a dynamic object based on dynamic contour key point tracking also includes: optimizing the motion pose of the dynamic object in each video clip in the video data using a bundle adjustment optimization framework based on dynamic contour key point tracking to obtain the motion trajectory of the dynamic object in the video data.

[0066] The above-mentioned dynamic object pose estimation method based on dynamic contour key point tracking provided by the present invention can obtain semantic information in video data in real time through an instance segmentation network, and can screen out dynamic objects in video data; the initial pose estimation of dynamic objects based on the discrete sampling method ensures the robust pose initialization of dynamic objects in the world coordinate system when they are observed for the first time. The pixel tracking method based on dynamic contour key points establishes a long-term robust spatiotemporal association of dynamic object contour points in a 2D pixel coordinate system. The three-dimensional-perspective algorithm based on dynamic contour key point tracking provides a preliminary dynamic object pose estimation, and the subsequent bundle adjustment optimization framework based on contour key point tracking combines the object contour key points, the three-dimensional coordinates of the spatial point, the camera pose, and the three-dimensional-perspective algorithm motion object pose estimation to optimize the dynamic object pose, and obtain accurate object motion, thereby generating an accurate dynamic object motion trajectory. The above-mentioned method provided by the present invention constructs a long-term robust spatiotemporal association of dynamic object contour key points and performs high-precision dynamic object pose estimation.

[0067] Figure 2 It is a flow chart of a method for dynamic object pose estimation and trajectory generation based on dynamic contour key point tracking according to an embodiment of the present invention.

[0068] The following is a specific embodiment and combined with the attached Figure 2 The above method provided by the present invention is further described in detail.

[0069] like Figure 2 As shown, the above method provided by the present invention includes acquiring video data through a video acquisition device, preprocessing the video data (or video stream input), sampling-based motion initialization, object pose estimation based on pixel-level tracking of dynamic contour feature points, and a bundle adjustment optimization framework based on contour point tracking.

[0070] Among them, in the video stream input preprocessing, the dynamic objects in the video stream are mainly extracted through the instance segmentation network, and potential dynamic objects are screened out from the scene of the video stream; the object pose estimation based on the pixel-level tracking of dynamic contour feature points mainly performs contour point extraction, that is, semantic feature point extraction of the contour of the dynamic object in the scene, and pixel tracking, that is, using the pixel tracking method to track the contour key points of the dynamic object and establish a long-term stable spatiotemporal association of the contour key points, and dynamic object pose estimation, that is, the camera pose is estimated through the SLAM system based on the feature point method, and then the camera pose and the results of pixel tracking are used to estimate the 6DoF (six degrees of freedom) pose of the dynamic object using the PnP algorithm; the bundle adjustment optimization framework based on contour point tracking mainly refers to the back-end optimization, that is, by integrating the camera pose, dynamic object pose and three-dimensional coordinates of spatial points (including static feature points and dynamic feature points) into the same bundle adjustment optimization framework based on contour key point tracking to optimize the above three variables simultaneously.

[0071] According to an embodiment of the present invention, the above-mentioned method of performing instance segmentation on each frame of video data using an instance segmentation network to obtain an instance segmentation result of each frame of image, extracting dynamic objects from the instance segmentation result of each frame of image, and filtering the instance segmentation result of each frame of image according to predefined filtering conditions to obtain a mask of dynamic objects in each frame of image in the video data includes: acquiring video data through a video acquisition device, wherein the video acquisition device includes a monocular camera, a binocular camera and a depth camera; performing instance segmentation on each frame of video data using an instance segmentation network, and extracting dynamic objects from the instance segmentation result of each frame of image to obtain dynamic objects in each frame of image, wherein the instance segmentation network includes a SOLO instance segmentation network; determining a predefined filtering condition based on the overlap rate of object masks between adjacent frame images in the video data and the ratio of object pixels to full-image pixels in each frame of image; filtering out static objects and objects smaller than a preset size from the instance segmentation result of each frame of image according to the predefined filtering condition to obtain a mask of dynamic objects in each frame of image.

[0072] Among them, the above-mentioned video data acquisition device is not limited to a monocular camera, a binocular camera and a depth (RGB-D) camera. Those skilled in the art can adopt a suitable video data acquisition device according to actual needs, and the video data collected by the device is also applicable to the present invention.

[0073] The above instance segmentation network is not limited to the SOLO instance segmentation network, and those skilled in the art can select a suitable instance segmentation network according to actual needs.

[0074] According to an embodiment of the present invention, the above-mentioned obtaining of video data through the video acquisition device includes: preprocessing the video data acquired by the monocular camera through a monocular depth estimation network to obtain a depth map of each frame image in the video data.

[0075] The above embodiment involves operation S110 . The following further describes in detail the preprocessing process of video data involved in operation S110 through specific implementation methods and in conjunction with the accompanying drawings.

[0076] Figure 3 It is a schematic diagram of detection results of detecting dynamic objects in video data using an instance segmentation network according to an embodiment of the present invention.

[0077] The above method provided by the present invention supports the video stream data collected by monocular, binocular and depth (RGB-D) cameras. First, the camera is calibrated with the checkerboard calibration method to obtain the intrinsic matrix and distortion coefficient of each camera, and the internal parameters of the camera are stored at the same time. In view of the defect of scale uncertainty of monocular video stream, the present invention uses a monocular depth estimation network to generate a depth map of each frame of the picture during the input of the monocular video stream. Subsequently, the video stream enters the preprocessing thread, and the thread uses the SOLO instance segmentation network to extract potential dynamic objects in the scene. At the same time, by inputting the image collected by the camera sensor into the instance segmentation network, the mask information of the detected object is output, including the type of object and the pixel position occupied by the object. On the basis of instance segmentation, the present invention filters out static objects and small objects in the scene, such as cars parked on the roadside, according to the IoU of objects between frames and the proportion of their pixels to the full-image pixels, and outputs the mask of absolutely dynamic objects. Figure 3 The figure shows the dynamic object segmentation result of SOLO in a single frame of KITTI odometer sequence. In particular, the instance segmentation framework SOLO used in this solution is a fast and compact open source segmentation model, which can ensure the accuracy of instance segmentation while ensuring the real-time performance when the technical solution of the present invention is combined with the mainstream visual SLAM system based on the feature point method.

[0078] According to an embodiment of the present invention, the above-mentioned method of obtaining the fitting ground equation of the initial frame image in the current video clip by using the random sampling consensus algorithm, and generating the initial posture of the dynamic object in the initial frame image according to the fitting ground equation of the initial frame image and the mask of the dynamic object in the initial frame image includes: setting the pitch angle and roll angle of the moving object in the initial frame image according to preset values, and fitting the ground of the initial frame image by using the random sampling consensus algorithm to obtain the fitting ground equation of the initial frame image; adjusting the camera self-motion posture of the current video clip according to the fitting ground equation of the initial frame image, and setting the ground of the initial frame image based on the adjustment result and the world coordinate system; projecting the mask of the dynamic object in the initial frame image into the world coordinate system, and predicting the yaw angle of the dynamic object on the two-dimensional pixel plane using the projection result. Assume discrete sampling of an angle range, obtain discrete sampling results, calculate the minimum area bounding box of the discrete sampling results in the two-dimensional projection area, and use the minimum area bounding box of the discrete sampling results as a top view of the 3D bounding box of the dynamic object in the initial frame image to estimate the yaw angle, X-axis initial point, Y-axis initial point, length and width of the 3D bounding box of the dynamic object in the initial frame image; based on the constraint conditions between the yaw angle of the dynamic object in the initial frame image and the 3D bounding box of the dynamic object in the initial frame image, calculate the Z-axis initial point and height of the 3D bounding box of the dynamic object to obtain all parameter values ​​of the 3D bounding box of the dynamic object; based on all parameter values ​​of the 3D bounding box of the dynamic object in the initial frame image, generate the initial posture of the dynamic object in the initial frame image, wherein the initial posture of the dynamic object is a posture with six degrees of freedom.

[0079] The above embodiment involves operation S120. The following further describes in detail the motion initialization process based on discrete sampling involved in operation S120 through specific implementation modes and in conjunction with the accompanying drawings.

[0080] Figure 4 is a schematic diagram of a posture initialization process of a dynamic object based on discrete sampling according to an embodiment of the present invention.

[0081] Dynamic object pose initialization plays a vital role in accurately locating the position of dynamic objects in the global map when they are first observed. Most existing methods estimate the initial pose of dynamic objects through a network. However, learning-based methods are difficult to generalize between different scenarios and difficult to adapt to complex autonomous driving scenarios. Therefore, the present invention proposes a sampling-based motion initialization method, which accurately estimates the 6DoF (six degrees of freedom) pose when a dynamic object is first observed. The process of dynamic object initialization is as follows: Figure 4As shown. Since almost all dynamic objects in autonomous driving scenarios are located on the ground, the present invention first sets the pitch angle (pitch) and roll angle (roll) to zero, and then calculates the optimal yaw angle (yaw) by sampling. The present invention first uses the RANSAC algorithm (Random Sample Consensus) to fit the ground and outputs the fitted ground equation, as shown in formula (1):

[0082] n 1 x+n 2 y+n 3 z+d=0(1),

[0083] Where n = [n 1 ,n 2 ,n 3 ] is the normal vector of the fitted plane, and d is the height of the plane. According to the equation of the fitted ground, the current camera self-motion posture is adjusted to align the ground horizontally with the world coordinate system and the height is set to 0. Subsequently, the present invention projects the mask of the dynamic object obtained in step 1 into the world coordinate system, and performs discrete sampling of the yaw angle from 0 to 180 degrees on the two-dimensional pixel plane. For each sampling result, a minimum area bounding box of the two-dimensional projection area is calculated. Then, the bounding box is used as a top view of the 3D bounding box of the moving object, thereby estimating the parameters of the 3D bounding box [t x ,t y ,yaw,length,width]. Then, using the constraint that all 3D points in the yaw direction need to be within the 3D bounding box, the remaining two parameters [t z ,height] can also be calculated. Finally, the initial pose of the dynamic object is generated.

[0084] According to an embodiment of the present invention, the above-mentioned method of using the dynamic contour key point tracking method to extract semantic feature points on the contour of the dynamic object of the initial frame image to obtain the dynamic contour key points of the dynamic object of the initial frame image, and using the pixel tracking method to perform pixel-level tracking on the dynamic contour key points to obtain the tracking result of the dynamic contour key points of the non-initial frame image in the current video clip includes: using the dynamic contour key point tracking method to extract semantic feature points on the contour of the dynamic object of the initial frame image in the current video clip to obtain the dynamic contour key points of the dynamic object of the initial frame image; using the pixel tracking method to perform pixel-level tracking on each non-initial frame image to obtain the matching tracking result of the dynamic contour key points of each non-initial frame image; according to the preset key point screening conditions, using the random sampling consensus algorithm to screen the dynamic contour key points of the dynamic object of each non-initial frame image to obtain the screened dynamic contour key points of each non-initial frame image.

[0085] According to an embodiment of the present invention, the above-mentioned tracking results of dynamic contour key points based on non-initial frame images, using a perspective-3D algorithm to generate a preliminary estimate of the position and pose of the dynamic object in the current video clip includes: based on the correspondence between the dynamic contour key points between adjacent video frames in the current video clip, using a PnP algorithm to generate a preliminary estimate of the position and pose of the dynamic object in the current video clip.

[0086] In generating a preliminary estimation of the pose of a dynamic object in a current video clip, a StarMap neural network is used to obtain the dynamic contour key points of the dynamic object in the initial frame image in the current video clip, and a pixel tracking method is used to track the dynamic contour key points of the initial frame image at the pixel level to obtain the dynamic contour key points of the dynamic object in the next frame image adjacent to the initial frame, and a RANSAC (random sampling consensus algorithm) algorithm is used to screen the dynamic contour key points of the current non-initial frame image so as to remove the external points of the current non-initial frame image to obtain the screened dynamic contour key points; based on the corresponding relationship represented by the screened dynamic contour key points, a PnP algorithm is used to generate a preliminary estimation of the pose of the dynamic object in the current non-initial frame; the above-mentioned dynamic contour key point tracking operation, pixel-level tracking operation, key point screening and external point deletion operation, and the preliminary estimation operation of the pose of the dynamic object of the video frame are repeated for each non-initial frame image of the current video clip until a preliminary estimation of the pose of the dynamic object in each frame image of the current video clip is obtained.

[0087] The above embodiment involves operations S130 to S140. The object pose estimation process based on pixel-level tracking of dynamic contour feature points involved in operations S130 to S140 will be further described in detail below through a specific implementation method.

[0088] First, dynamic contour key point tracking enhances the key point representation of dynamic objects by introducing contour and semantic information, and is combined with pixel tracking methods to establish long-term robust spatiotemporal association of dynamic objects. Subsequently, based on the results of dynamic contour point tracking, the present invention uses a simple but effective PnP algorithm (i.e., a three-dimensional-perspective algorithm) to obtain a preliminary estimate of the dynamic object's pose. The PnP algorithm is a method for estimating 6DoF motion based on matching points, and is mainly used to solve the target motion when the 2D-3D point correspondence of consecutive frames is known. Among them, P3P only needs three 2D-3D matching point pairs to estimate the motion pose. Thanks to the high precision of dynamic object contour feature point tracking, the present invention estimates that at least four 2D-3D correspondences are required for the motion of dynamic objects, in which an additional pair of matching points is used to verify the correctness of the estimated pose. The present invention evenly extracts ten key points on each detected dynamic object contour, and uses RANSAC to eliminate outliers and obtain a preliminary estimate of the motion pose of the moving object. In addition, the preliminary estimated target object pose is optimized by minimizing the reprojection error cost function. After applying the PnP algorithm to generate a preliminary motion estimate, the key frame I k The reprojection error of a contour key point can be expressed by formula (2):

[0089]

[0090] in, The present invention uses Lie algebra Parameterize and combine the dynamic contour key points and dynamic object motion estimation in the local map. The cost function at this time is shown in formula (3):

[0091]

[0092] Where m is the number of point pairs of dynamic contour feature matching points. Due to the existence of occlusion, the number of contour feature points will gradually decrease during camera movement, resulting in pixel tracking failure. To ensure the robustness of the P3P Solver, when the number of currently tracked dynamic contour key points decreases, the above algorithm provided by the present invention will detect new dynamic contour key points in the current frame and set the current frame as a map key frame. Subsequently, the new key frame is added to the map and the feature points are updated.

[0093] According to an embodiment of the present invention, the above-mentioned dynamic adjustment and optimization of the factor graph to generate the dynamic pose of the dynamic object in the current video includes: based on the tracking results of the dynamic contour key points of each non-initial frame image, detecting the change in the number of dynamic contour key points of the current non-initial frame image to obtain the detection result; when the detection result meets the predefined change condition, obtaining new dynamic contour key points from the current non-initial frame image, and setting the current non-initial frame image as the map key frame image of the current video clip; adding the map key frame image to the map of the current video clip and updating the dynamic contour key points of the map key frame image; based on the reprojection error of the map key frame image, using Lie algebra to parameterize the rigid body transformation of the dynamic object in the map key frame image to obtain the reprojection error cost function; fixing the estimation result of the camera self-motion pose of the current video clip, using the heuristic algorithm to adjust the weights in the factor graph, and optimizing the preliminary estimation of the pose of the dynamic object of each non-initial frame image by minimizing the reprojection error cost function to generate the dynamic pose of the dynamic object in the current video.

[0094] The above embodiment involves operation S150. The following further describes in detail the cluster adjustment optimization framework based on dynamic contour key point tracking involved in operation S150 through specific implementation modes and in conjunction with the accompanying drawings.

[0095] Figure 5 is a schematic diagram of a bundle adjustment optimization framework based on dynamic contour key point tracking according to an embodiment of the present invention.

[0096] The backend of the SLAM problem is usually modeled as a factor graph optimization problem to optimize and improve the accuracy of camera and object motion estimation. Based on the contour key point tracking method, the present invention designs a bundle adjustment optimization framework based on contour point tracking to further improve the accuracy of dynamic object motion estimation in the local map. Figure 5 As shown in the figure, in order to better express the dynamic scene, the present invention first fixes the camera posture, and then constructs a sliding window internal factor graph to optimize the motion estimation of dynamic objects. In addition, this framework will call the sampling-based motion initialization method again every 5 key frames to correct the posture of the current frame so that it remains horizontal with the world coordinate system and has a height of zero. Finally, all dynamic contour key points, camera self-motion, dynamic object motion, and sampling-based dynamic object motion are constructed into Figure 5 The factor graph is shown, and the motion of the moving object is optimized using the factor graph. Since the dynamic object motion obtained by the heuristic method (the dynamic object motion initialization method based on sampling) usually has a higher accuracy, the invention increases the weight of the object motion obtained by the sampling method in the factor graph (i.e. Figure 5 The sampling factor in ) is used to obtain better optimization effect.

[0097] Optionally, the SLAM system selects the mainstream open source feature point-based visual SLAM system ORB-SLAM2, and the input can be monocular, binocular, or RGB-D video stream. For monocular input, a monocular depth prediction network can be used to generate a depth map for each frame, and then the robot's self-motion is estimated in the front-end odometer of the SLAM system based on the depth map and video stream. At the same time, a local map is constructed based on multiple frames, and the robot's self-motion within the local map is optimized.

[0098] Optionally, the object contour key points P are combined in the back-end bundle adjustment (BA) framework of the visual SLAM system o , static feature point P w , camera pose T ego And the pose T of the moving object calculated by PnP Solver dyn Joint optimization. This framework constructs a factor graph optimization model based on the constraint that the three-dimensional coordinates of the contour feature points in the object coordinate system remain unchanged. ego The remaining three variables are jointly optimized, and the sparsity of the Hessian matrix is ​​used to accelerate the calculation and reduce the computational complexity.

[0099] The odometer front end of the present invention integrates the instance segmentation network to extract semantic information from the video stream under the premise of ensuring the real-time performance of the system, and screens out the absolute dynamic objects in the scene according to the heuristic settings. The SLAM front end performs high-precision robot self-motion pose estimation based on the refined static background. The dynamic object pose estimation method based on sampling ensures the robust pose initialization of the dynamic object in the world coordinate system when it is first observed. The pixel tracking method based on the contour key point establishes a long-term robust dynamic object contour point spatiotemporal association in the 2D pixel coordinate system. The PnP Solver based on contour key point tracking provides a preliminary dynamic object pose estimation, and the subsequent bundle adjustment optimization framework based on contour key point tracking combines the object contour key points, the three-dimensional coordinates of the space point, the camera pose and the PnP Solver motion object pose estimation to optimize the dynamic object pose, obtain accurate object motion, and thus generate an accurate dynamic object motion trajectory. This scheme constructs a long-term robust dynamic object contour key point spatiotemporal association and performs high-precision dynamic object pose estimation.

[0100] Figure 6 A block diagram of an electronic device suitable for implementing a method for estimating a posture of a dynamic object based on dynamic contour key point tracking according to an embodiment of the present invention is schematically shown.

[0101] like Figure 6As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0102] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in one or more memories.

[0103] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0104] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0105] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0106] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0107] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for estimating the pose of a dynamic object based on dynamic contour key point tracking, characterized in that: include: Perform instance segmentation on each frame of the video data using an instance segmentation network to obtain an instance segmentation result for each frame of the image, extract dynamic objects from the instance segmentation result for each frame of the image, and filter the instance segmentation result for each frame of the image according to a predefined filtering condition to obtain a mask of the dynamic object for each frame of the image in the video data; A fitting ground equation of an initial frame image in a current video segment is obtained by using a random sampling consensus algorithm, and an initial posture of a dynamic object in the initial frame image is generated according to the fitting ground equation of the initial frame image and a mask of a dynamic object in the initial frame image, wherein the initial frame image represents a frame image in which a dynamic object is first observed in the current video segment, and the current video segment represents continuous video frames intercepted from the video data according to predefined conditions; Extracting semantic feature points on the contour of the dynamic object in the initial frame image by using a dynamic contour key point tracking method to obtain dynamic contour key points of the dynamic object in the initial frame image, and tracking the dynamic contour key points at the pixel level by using a pixel tracking method to obtain tracking results of dynamic contour key points of non-initial frame images in the current video clip; Based on the tracking results of the dynamic contour key points of the non-initial frame image, a perspective-three-dimensional algorithm is used to generate a preliminary estimate of the position and posture of the dynamic object in the current video clip, and an estimate of the self-motion position and posture of the camera in the current video clip is obtained by using image positioning and mapping technology; A factor graph of the dynamic objects in the current video clip is constructed using the dynamic contour key points of the current video clip, the initial posture of the dynamic objects, the preliminary estimation of the posture of the dynamic objects, and the estimation results of the camera's self-motion posture, and the dynamic posture of the dynamic objects in the current video is generated by dynamically adjusting and optimizing the factor graph.

2. The method according to claim 1, characterized in that Also includes: The motion pose of the dynamic object in each video clip in the video data is optimized by using a bundle adjustment optimization framework based on dynamic contour key point tracking to obtain the motion trajectory of the dynamic object in the video data.

3. The method according to claim 1, characterized in that Performing instance segmentation on each frame of the video data using an instance segmentation network to obtain an instance segmentation result of each frame of the image, extracting dynamic objects from the instance segmentation result of each frame of the image, and filtering the instance segmentation result of each frame of the image according to a predefined filtering condition to obtain a mask of the dynamic object of each frame of the image in the video data includes: Acquiring the video data through a video acquisition device, wherein the video acquisition device includes a monocular camera, a binocular camera, and a depth camera; Performing instance segmentation on each frame of the video data using the instance segmentation network, and extracting dynamic objects from the instance segmentation results of each frame of the image to obtain the dynamic objects of each frame of the image, wherein the instance segmentation network includes a SOLO instance segmentation network; Determining a predefined screening condition according to an overlap rate of object masks between adjacent frame images in the video data and a ratio of object pixels to full image pixels in each frame image; According to the predefined screening condition, static objects and objects smaller than a preset size are filtered out from the instance segmentation results of each frame of the image, so as to obtain a mask of the dynamic objects of each frame of the image.

4. The method according to claim 3, characterized in that Acquiring the video data by a video acquisition device includes: The video data collected by the monocular camera is preprocessed through a monocular depth estimation network to obtain a depth map of each frame image in the video data.

5. The method according to claim 1, characterized in that Obtaining a fitting ground equation of an initial frame image in a current video clip by using a random sampling consensus algorithm, and generating an initial posture of a dynamic object of the initial frame image according to the fitting ground equation of the initial frame image and a mask of the dynamic object of the initial frame image includes: The pitch angle and roll angle of the moving object in the initial frame image are set according to preset values, and the ground of the initial frame image is fitted using a random sampling consensus algorithm to obtain a fitting ground equation of the initial frame image; According to the fitted ground equation of the initial frame image, the camera self-motion posture of the current video clip is adjusted, and the ground of the initial frame image is set based on the adjustment result and the world coordinate system; The mask of the dynamic object in the initial frame image is projected into the world coordinate system, and the yaw angle of the dynamic object is discretely sampled within a preset angle range on a two-dimensional pixel plane using the projection result to obtain a discrete sampling result. Calculate the minimum area bounding box of the discrete sampling result in the two-dimensional projection area, and use the minimum area bounding box of the discrete sampling result as a top view of the 3D bounding box of the dynamic object of the initial frame image to estimate the yaw angle, X-axis initial point, Y-axis initial point, length and width of the 3D bounding box of the dynamic object of the initial frame image; Based on the constraint condition between the yaw angle of the dynamic object in the initial frame image and the 3D bounding box of the dynamic object in the initial frame image, calculating the Z-axis initial point and height of the 3D bounding box of the dynamic object to obtain all parameter values ​​of the 3D bounding box of the dynamic object; An initial posture of the dynamic object in the initial frame image is generated based on all parameter values ​​of the 3D bounding box of the dynamic object in the initial frame image, wherein the initial posture of the dynamic object is a posture with six degrees of freedom.

6. The method according to claim 1, characterized in that Extracting semantic feature points on the contour of the dynamic object in the initial frame image by using a dynamic contour key point tracking method to obtain dynamic contour key points of the dynamic object in the initial frame image, and tracking the dynamic contour key points at the pixel level by using a pixel tracking method to obtain tracking results of dynamic contour key points of the non-initial frame image in the current video clip, including: Extracting semantic feature points on the contour of the dynamic object in the initial frame image in the current video clip using the dynamic contour key point tracking method to obtain the dynamic contour key points of the dynamic object in the initial frame image; Perform pixel-level tracking on each of the non-initial frame images using the pixel tracking method to obtain matching tracking results of dynamic contour key points of each of the non-initial frame images; According to the preset key point screening conditions, the dynamic contour key points of the dynamic objects in each of the non-initial frame images are screened using the random sampling consensus algorithm to obtain the screened dynamic contour key points of each of the non-initial frame images.

7. The method according to claim 1, characterized in that Generating the dynamic pose of the dynamic object in the current video by dynamically adjusting and optimizing the factor graph includes: Based on the tracking results of the dynamic contour key points of each non-initial frame image, detecting the change in the number of dynamic contour key points of the current non-initial frame image to obtain a detection result; When the detection result satisfies a predefined change condition, acquiring new dynamic contour key points from the current non-initial frame image, and setting the current non-initial frame image as a map key frame image of the current video segment; Adding the map key frame image to the map of the current video clip and updating the dynamic contour key points of the map key frame image; Based on the reprojection error of the map key frame image, the rigid body transformation of the dynamic object in the map key frame image is parameterized by using Lie algebra to obtain a reprojection error cost function; The estimated result of the camera self-motion pose of the current video clip is fixed, the weights in the factor graph are adjusted using a heuristic algorithm, and the preliminary estimate of the pose of the dynamic object in each non-initial frame image is optimized by minimizing the reprojection error cost function to generate the dynamic pose of the dynamic object in the current video.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 7.

Images