A dynamic object pose estimation method based on dynamic contour key point tracking

Abstract

The present invention provides a method for estimating the pose of a dynamic object based on dynamic contour key point tracking. The method comprises: extracting dynamic objects from video data using an instance segmentation network and obtaining a mask of the dynamic object; generating an initial pose of the dynamic object in an initial frame image in a current video clip based on a discrete sampling method; tracking the dynamic object in non-initial frame images in the current video clip using a dynamic contour key point tracking method, and tracking the dynamic contour key points at the pixel level using a pixel tracking method; generating a preliminary pose estimate of the dynamic object in the frame image in the current video clip using a PnP algorithm, and obtaining an estimated result of the camera's self-motion pose in the current video clip using image localization and mapping technology; and generating and optimizing the dynamic pose of the dynamic object in the current video clip using the dynamic contour key points, the initial pose of the dynamic object, the preliminary pose estimate of the dynamic object, and the estimated result of the camera's self-motion pose in the current video clip.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a dynamic object posture estimation method based on dynamic contour key point tracking, an electronic device, and a storage medium. Background Art

[0002] With the advancement of technology, robots are becoming increasingly intertwined with people's lives. Simultaneous localization and mapping (SLAM) technology is the core technology for robots to autonomously explore and navigate their environments. Most existing SLAM technologies assume that the scene is absolutely static, an assumption that cannot meet the application requirements of real-world scenarios. Therefore, both industry and academia are in urgent need of 4D SLAM (3D+time), or dynamic SLAM, especially in terms of scene understanding and task decision-making. The current difficulty with dynamic (4D) SLAM lies in the inability to establish long-term, stable spatiotemporal information associations for dynamic objects in the scene, which is crucial for dynamic object tracking and the application of SLAM systems in real-world scenarios. Therefore, existing visual SLAM still faces the following challenges: It is impossible to establish long-term, robust spatiotemporal information associations for dynamic objects; and existing 6DoF pose estimation methods for dynamic objects have high computational complexity and cannot be well integrated with feature point-based SLAM systems. Summary of the Invention

[0003] In view of the above problems, the present invention provides a dynamic object pose estimation method based on dynamic contour key point tracking, an electronic device and a storage medium, in order to solve at least one of the above problems.

[0004] According to a first aspect of the present invention, a method for estimating the pose of a dynamic object based on dynamic contour key point tracking is provided, comprising:

[0005] Utilize the instance segmentation network to perform instance segmentation on each frame of the video data to obtain the instance segmentation results of each frame, extract dynamic objects from the instance segmentation results of each frame, and filter the instance segmentation results of each frame according to predefined filtering conditions to obtain the mask of the dynamic objects in each frame of the video data;

[0006] A random sampling consensus algorithm is used to obtain a fitting ground equation of an initial frame image in the current video clip, and an initial pose of the dynamic object in the initial frame image is generated based on the fitting ground equation of the initial frame image and a mask of the dynamic object in the initial frame image, wherein the initial frame image represents a frame image in which the dynamic object is first observed in the current video clip, and the current video clip represents a continuous video frame intercepted from the video data according to predefined conditions;

[0007] Extracting semantic feature points on the contour of the dynamic object in the initial frame image using a dynamic contour key point tracking method to obtain dynamic contour key points of the dynamic object in the initial frame image, and tracking the dynamic contour key points at the pixel level using a pixel tracking method to obtain tracking results of the dynamic contour key points of the non-initial frame image in the current video clip;

[0008] Based on the tracking results of dynamic contour key points in non-initial frame images, a perspective-3D algorithm is used to generate a preliminary estimate of the pose of the dynamic object in the current video clip, and image positioning and mapping technology is used to obtain the estimated result of the camera's self-motion pose in the current video clip;

[0009] The factor graph of the dynamic objects in the current video clip is constructed using the dynamic contour key points of the current video clip, the initial pose of the dynamic object, the preliminary estimation of the dynamic object's pose and the estimation results of the camera's self-motion pose. The dynamic pose of the dynamic object in the current video is generated by dynamically adjusting and optimizing the factor graph.

[0010] According to an embodiment of the present invention, the above-mentioned method for estimating the pose of a dynamic object based on dynamic contour key point tracking further includes:

[0011] The motion pose of dynamic objects in each video clip in the video data is optimized using a bundle adjustment optimization framework based on dynamic contour key point tracking to obtain the motion trajectory of the dynamic objects in the video data.

[0012] According to an embodiment of the present invention, the above-mentioned instance segmentation network is used to perform instance segmentation on each frame of the video data to obtain an instance segmentation result for each frame of the image, extracting dynamic objects from the instance segmentation result of each frame of the image, and filtering the instance segmentation result of each frame of the image according to a predefined filtering condition to obtain a mask of the dynamic object in each frame of the video data.

[0013] Acquire video data through a video acquisition device, wherein the video acquisition device includes a monocular camera, a binocular camera, and a depth camera;

[0014] Perform instance segmentation on each frame of the video data using an instance segmentation network, and extract dynamic objects from the instance segmentation results of each frame to obtain the dynamic objects in each frame, wherein the instance segmentation network includes a SOLO instance segmentation network;

[0015] Determine predefined screening conditions based on the overlap rate of object masks between adjacent frame images in the video data and the ratio of object pixels to full image pixels in each frame image;

[0016] According to predefined filtering conditions, static objects and objects smaller than a preset size are filtered out from the instance segmentation results of each frame image to obtain the mask of dynamic objects in each frame image.

[0017] According to an embodiment of the present invention, obtaining video data through a video acquisition device includes:

[0018] The video data collected by the monocular camera is preprocessed through a monocular depth estimation network to obtain a depth map of each frame image in the video data.

[0019] According to an embodiment of the present invention, the above-mentioned obtaining the fitting ground equation of the initial frame image in the current video clip by using the random sampling consensus algorithm, and generating the initial pose of the dynamic object in the initial frame image based on the fitting ground equation of the initial frame image and the mask of the dynamic object in the initial frame image includes:

[0020] According to preset values, the pitch angle and roll angle of the moving object in the initial frame image are set, and the ground of the initial frame image is fitted using a random sampling consensus algorithm to obtain a fitting ground equation of the initial frame image;

[0021] Adjust the camera's self-motion pose for the current video clip according to the ground equation fitted to the initial frame image, and set the ground of the initial frame image based on the adjustment result and the world coordinate system;

[0022] The mask of the dynamic object in the initial frame image is projected into the world coordinate system, and the projection result is used to perform discrete sampling of the yaw angle of the dynamic object within a preset angle range on the two-dimensional pixel plane to obtain the discrete sampling result.

[0023] Calculate the minimum area bounding box of the discrete sampling result in the two-dimensional projection area, and use the minimum area bounding box of the discrete sampling result as the top view of the 3D bounding box of the dynamic object in the initial frame image to estimate the yaw angle, X-axis initial point, Y-axis initial point, length and width of the 3D bounding box of the dynamic object in the initial frame image;

[0024] Based on the constraint condition between the yaw angle of the dynamic object in the initial frame image and the 3D bounding box of the dynamic object in the initial frame image, the Z-axis initial point and height of the 3D bounding box of the dynamic object are calculated to obtain all parameter values ​​of the 3D bounding box of the dynamic object;

[0025] An initial posture of the dynamic object in the initial frame image is generated based on all parameter values ​​of the 3D bounding box of the dynamic object in the initial frame image, wherein the initial posture of the dynamic object is a posture with six degrees of freedom.

[0026] According to an embodiment of the present invention, the above-mentioned dynamic contour key point tracking method is used to extract semantic feature points on the contour of the dynamic object in the initial frame image to obtain the dynamic contour key points of the dynamic object in the initial frame image, and the pixel tracking method is used to track the dynamic contour key points at the pixel level to obtain the tracking results of the dynamic contour key points of the non-initial frame image in the current video clip, including:

[0027] Extracting semantic feature points on the contour of the dynamic object in the initial frame image of the current video clip using a dynamic contour key point tracking method to obtain dynamic contour key points of the dynamic object in the initial frame image;

[0028] Utilize the pixel tracking method to track each non-initial frame image at the pixel level, and obtain the matching tracking results of the dynamic contour key points of each non-initial frame image;

[0029] According to the preset key point screening conditions, the dynamic contour key points of the dynamic object in each non-initial frame image are screened using a random sampling consensus algorithm to obtain the screened dynamic contour key points of each non-initial frame image.

[0030] According to an embodiment of the present invention, the above-mentioned generating the dynamic pose of the dynamic object in the current video by dynamically adjusting and optimizing the factor graph includes:

[0031] Based on the tracking results of the dynamic contour key points of each non-initial frame image, detecting the change in the number of the dynamic contour key points of the current non-initial frame image to obtain a detection result;

[0032] When the detection result satisfies the predefined change condition, new dynamic contour key points are obtained from the current non-initial frame image, and the current non-initial frame image is set as the map key frame image of the current video segment;

[0033] Adding a map keyframe image to the map of the current video segment and updating the dynamic contour key points of the map keyframe image;

[0034] Based on the reprojection error of the map key frame image, the rigid body transformation of the dynamic objects in the map key frame image is parameterized using Lie algebra to obtain the reprojection error cost function;

[0035] The estimated result of the camera's self-motion pose of the current video clip is fixed, the weights in the factor graph are adjusted using a heuristic algorithm, and the preliminary estimate of the pose of the dynamic object in each non-initial frame image is optimized by minimizing the reprojection error cost function to generate the dynamic pose of the dynamic object in the current video.

[0036] According to a second aspect of the present invention, there is provided an electronic device, comprising:

[0037] one or more processors;

[0038] a storage device for storing one or more programs,

[0039] When one or more programs are executed by one or more processors, the one or more processors execute a dynamic object pose estimation method based on dynamic contour key point tracking.

[0040] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes a dynamic object pose estimation method based on dynamic contour key point tracking.

[0041] The above-mentioned dynamic object pose estimation method based on dynamic contour key point tracking provided by the present invention can obtain semantic information in video data in real time through an instance segmentation network, and can screen out dynamic objects in the video data; the initial pose estimation of dynamic objects based on the discrete sampling method ensures the robust pose initialization of the dynamic object in the world coordinate system when it is first observed. The pixel tracking method based on dynamic contour key points establishes a long-term robust spatiotemporal association of dynamic object contour points in a 2D pixel coordinate system. The three-dimensional-perspective algorithm based on dynamic contour key point tracking provides a preliminary dynamic object pose estimation, and the subsequent bundle adjustment optimization framework based on contour key point tracking combines the object contour key points, the three-dimensional coordinates of the spatial point, the camera pose and the three-dimensional-perspective algorithm motion object pose estimation to optimize the dynamic object pose, obtain accurate object motion, and thus generate an accurate dynamic object motion trajectory. The above-mentioned method provided by the present invention constructs a long-term robust spatiotemporal association of dynamic object contour key points and performs high-precision dynamic object pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of a method for estimating the pose of a dynamic object based on dynamic contour key point tracking according to an embodiment of the present invention;

[0043] Figure 2 is a flow chart of a method for dynamic object pose estimation and trajectory generation based on dynamic contour key point tracking according to an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of detection results of dynamic objects in video data using an instance segmentation network according to an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of a posture initialization process of a dynamic object based on discrete sampling according to an embodiment of the present invention;

[0046] Figure 5is a schematic diagram of a bundle adjustment optimization framework based on dynamic contour key point tracking according to an embodiment of the present invention;

[0047] Figure 6 A block diagram of an electronic device suitable for implementing a method for estimating the pose of a dynamic object based on dynamic contour key point tracking according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0049] In order to overcome the technical problems existing in the existing technical solutions, such as the defects of SLAM technology based on static scene assumptions in real-world scene applications, the present invention provides a pose estimation method based on tracking the key points of the dynamic object contour, which realizes the establishment of long-term robust spatiotemporal information association for dynamic objects in the scene and performs high-precision online trajectory estimation and generation; the solution fully considers the real-time and high-precision requirements of robot operation in dynamic scenes, and designs a multi-dynamic target tracking and trajectory estimation method with strong portability, low cost and high accuracy; at the same time, the method also has the characteristics of wide applicability and strong portability.

[0050] Figure 1 4 is a flow chart of a method for estimating the pose of a dynamic object based on dynamic contour key point tracking according to an embodiment of the present invention.

[0051] like Figure 1 As shown, the above-mentioned method for estimating the pose of a dynamic object based on dynamic contour key point tracking includes operations S110 to S150.

[0052] In operation S110, each frame image of the video data is instance segmented using an instance segmentation network to obtain an instance segmentation result for each frame image, dynamic objects are extracted from the instance segmentation result for each frame image, and the instance segmentation result for each frame image is filtered according to predefined filtering conditions to obtain a mask of the dynamic object for each frame image in the video data.

[0053] The above predefined filtering conditions mean that based on instance segmentation, static objects and small objects in the scene are filtered out according to the Intersection over Union (IoU) of objects between frames and the ratio of their pixels to the pixels of the full image.

[0054] In operation S120, a random sampling consensus algorithm is used to obtain a fitting ground equation of the initial frame image in the current video clip, and an initial pose of the dynamic object in the initial frame image is generated according to the fitting ground equation of the initial frame image and the mask of the dynamic object in the initial frame image.

[0055] The initial frame image represents a frame image in which a dynamic object is first observed in the current video segment, and the current video segment represents continuous video frames captured from video data according to predefined conditions.

[0056] The above predefined condition means that the frame image currently being processed is used as a node, and multiple consecutive frame images before the current frame image are intercepted from the video data to obtain the current video segment.

[0057] For example, if the currently processed frame image is the 50th frame image of the video data, the current video segment is obtained by intercepting a plurality of consecutive frame images (eg, 10 consecutive frame images) before the 50th frame image.

[0058] The initial posture of the dynamic object has six degrees of freedom (6DoF).

[0059] In operation S130, the semantic feature points on the contour of the dynamic object in the initial frame image are extracted using a dynamic contour key point tracking method to obtain the dynamic contour key points of the dynamic object in the initial frame image, and the dynamic contour key points are tracked at the pixel level using a pixel tracking method to obtain the tracking results of the dynamic contour key points of the non-initial frame image in the current video clip.

[0060] In operation S140, based on the tracking results of the dynamic contour key points of the non-initial frame image, a perspective-3D algorithm is used to generate a preliminary estimate of the position of the dynamic object in the current video clip, and the image positioning and mapping technology is used to obtain the estimated result of the camera self-motion pose in the current video clip.

[0061] The aforementioned image positioning and mapping technology (Simultaneous Localization and Mapping) is an algorithm for simultaneous visual positioning and map construction. It extracts feature information from images and uses methods such as feature matching and posture calculation to achieve camera positioning and map construction.

[0062] The above tracking results represent the corresponding relationship between the dynamic contour key points between two adjacent frames in the current video clip.

[0063] The above perspective-three-dimensional algorithm represents the PnP (Perspective-n-Point) algorithm, which is used to solve the correspondence between 3D points and 2D points. Those skilled in the art can adopt other algorithms for solving the correspondence between 3D points and 2D points according to actual needs.

[0064] In operation S150, a factor graph of the dynamic object in the current video clip is constructed using the dynamic contour key points of the current video clip, the initial pose of the dynamic object, the preliminary estimation of the posture of the dynamic object, and the estimation results of the camera's self-motion posture. The dynamic posture of the dynamic object in the current video is generated by dynamically adjusting and optimizing the factor graph.

[0065] According to an embodiment of the present invention, the above-mentioned method for estimating the pose of a dynamic object based on dynamic contour key point tracking also includes: optimizing the motion pose of the dynamic object in each video clip in the video data using a bundle adjustment optimization framework based on dynamic contour key point tracking to obtain the motion trajectory of the dynamic object in the video data.

[0066] The above-mentioned dynamic object pose estimation method based on dynamic contour key point tracking provided by the present invention can obtain semantic information in video data in real time through an instance segmentation network, and can screen out dynamic objects in the video data; the initial pose estimation of dynamic objects based on the discrete sampling method ensures the robust pose initialization of the dynamic object in the world coordinate system when it is first observed. The pixel tracking method based on dynamic contour key points establishes a long-term robust spatiotemporal association of dynamic object contour points in a 2D pixel coordinate system. The three-dimensional-perspective algorithm based on dynamic contour key point tracking provides a preliminary dynamic object pose estimation, and the subsequent bundle adjustment optimization framework based on contour key point tracking combines the object contour key points, the three-dimensional coordinates of the spatial point, the camera pose and the three-dimensional-perspective algorithm motion object pose estimation to optimize the dynamic object pose, obtain accurate object motion, and thus generate an accurate dynamic object motion trajectory. The above-mentioned method provided by the present invention constructs a long-term robust spatiotemporal association of dynamic object contour key points and performs high-precision dynamic object pose estimation.

[0067] Figure 2 4 is a flow chart of a method for dynamic object pose estimation and trajectory generation based on dynamic contour key point tracking according to an embodiment of the present invention.

[0068] The following is a detailed description of the embodiments and the attached Figure 2 The above method provided by the present invention is further described in detail.

[0069] like Figure 2 As shown, the above-mentioned method provided by the present invention includes obtaining video data through a video acquisition device, preprocessing of video data (or video stream input), motion initialization based on sampling, object pose estimation based on pixel-level tracking of dynamic contour feature points, and a bundle adjustment optimization framework based on contour point tracking.

[0070] Among them, in the video stream input preprocessing, dynamic objects are mainly extracted from the video stream through the instance segmentation network, and potential dynamic objects are screened out from the scene of the video stream; object pose estimation based on pixel-level tracking of dynamic contour feature points mainly performs contour point extraction, that is, semantic feature point extraction of dynamic object contours in the scene, and pixel tracking, that is, using the pixel tracking method to track the contour key points of dynamic objects and establish long-term stable spatiotemporal associations of contour key points, and dynamic object pose estimation, that is, estimating the camera pose through the SLAM system based on the feature point method, and then using the camera pose and the results of pixel tracking to estimate the 6DoF (six degrees of freedom) pose of the dynamic object using the PnP algorithm; the bundle adjustment optimization framework based on contour point tracking mainly refers to back-end optimization, that is, by integrating the camera pose, dynamic object pose and three-dimensional coordinates of spatial points (including static feature points and dynamic feature points) into the same bundle adjustment optimization framework based on contour key point tracking to simultaneously optimize the above three variables.

[0071] According to an embodiment of the present invention, the above-mentioned method of performing instance segmentation on each frame of video data using an instance segmentation network to obtain an instance segmentation result for each frame of image, extracting dynamic objects from the instance segmentation result of each frame of image, and filtering the instance segmentation result of each frame of image according to predefined filtering conditions to obtain a mask of dynamic objects in each frame of image in the video data includes: acquiring video data through a video acquisition device, wherein the video acquisition device includes a monocular camera, a binocular camera and a depth camera; performing instance segmentation on each frame of video data using an instance segmentation network, and extracting dynamic objects from the instance segmentation result of each frame of image to obtain dynamic objects in each frame of image, wherein the instance segmentation network includes a SOLO instance segmentation network; determining a predefined filtering condition based on the overlap rate of object masks between adjacent frame images in the video data and the ratio of object pixels to full-image pixels in each frame of image; filtering out static objects and objects smaller than a preset size from the instance segmentation result of each frame of image according to the predefined filtering condition to obtain a mask of dynamic objects in each frame of image.

[0072] Among them, the above-mentioned video data acquisition device is not limited to a monocular camera, a binocular camera and a depth (RGB-D) camera. Those skilled in the art can adopt a suitable video data acquisition device according to actual needs, and the video data collected by it is also applicable to the present invention.

[0073] The above instance segmentation network is not limited to the SOLO instance segmentation network, and those skilled in the art can select a suitable instance segmentation network according to actual needs.

[0074] According to an embodiment of the present invention, the above-mentioned acquisition of video data through the video acquisition device includes: preprocessing the video data acquired by the monocular camera through a monocular depth estimation network to obtain a depth map of each frame image in the video data.

[0075] The above embodiment involves operation S110 . The following further describes the pre-processing process of video data involved in operation S110 in detail through specific implementations and in conjunction with the accompanying drawings.

[0076] Figure 3 2 is a schematic diagram of detection results of dynamic objects in video data using an instance segmentation network according to an embodiment of the present invention.

[0077] The above method provided by the present invention supports video stream data collected by monocular, binocular and depth (RGB-D) cameras. First, the camera is calibrated with the checkerboard calibration method to obtain the intrinsic matrix and distortion coefficient of each camera, and the internal parameters of the camera are stored at the same time. In view of the defect of scale uncertainty of monocular video stream, the present invention uses a monocular depth estimation network to generate a depth map of each frame during the input of the monocular video stream. Subsequently, the video stream enters the preprocessing thread, and the thread uses the SOLO instance segmentation network to extract potential dynamic objects in the scene. At the same time, by inputting the image collected by the camera sensor into the instance segmentation network, the mask information of the detected object is output, including the type of object and the pixel position occupied by the object. Based on instance segmentation, the present invention filters out static objects and small objects in the scene, such as cars parked on the roadside, according to the inter-frame object IoU and the ratio of its pixels to the full-image pixels, and outputs the mask of the absolute dynamic object. Figure 3 The figure shows the dynamic object segmentation results of SOLO for a single frame of a KITTI odometry sequence. Specifically, the instance segmentation framework SOLO, utilized in this solution, is a fast and compact open-source segmentation model. It maintains instance segmentation accuracy while also ensuring real-time performance when combined with mainstream feature-based visual SLAM systems.

[0078] According to an embodiment of the present invention, the above-mentioned method of obtaining the fitting ground equation of the initial frame image in the current video clip by using the random sampling consensus algorithm, and generating the initial posture of the dynamic object in the initial frame image based on the fitting ground equation of the initial frame image and the mask of the dynamic object in the initial frame image includes: setting the pitch angle and roll angle of the moving object in the initial frame image according to the preset value, and fitting the ground of the initial frame image by using the random sampling consensus algorithm to obtain the fitting ground equation of the initial frame image; adjusting the camera self-motion posture of the current video clip according to the fitting ground equation of the initial frame image, and setting the ground of the initial frame image based on the adjustment result and the world coordinate system; projecting the mask of the dynamic object in the initial frame image into the world coordinate system, and using the projection result to predict the yaw angle of the dynamic object on the two-dimensional pixel plane. Assume discrete sampling of an angle range, obtain discrete sampling results, calculate the minimum area bounding box of the discrete sampling results in the two-dimensional projection area, and use the minimum area bounding box of the discrete sampling results as a top view of the 3D bounding box of the dynamic object in the initial frame image to estimate the yaw angle, X-axis initial point, Y-axis initial point, length and width of the 3D bounding box of the dynamic object in the initial frame image; based on the constraints between the yaw angle of the dynamic object in the initial frame image and the 3D bounding box of the dynamic object in the initial frame image, calculate the Z-axis initial point and height of the 3D bounding box of the dynamic object to obtain all parameter values ​​of the 3D bounding box of the dynamic object; based on all parameter values ​​of the 3D bounding box of the dynamic object in the initial frame image, generate the initial posture of the dynamic object in the initial frame image, wherein the initial posture of the dynamic object is a posture with six degrees of freedom.

[0079] The above embodiment involves operation S120 . The following further describes the discrete sampling-based motion initialization process involved in operation S120 in detail through specific implementations and in conjunction with the accompanying drawings.

[0080] Figure 4 3 is a schematic diagram of a posture initialization process of a dynamic object based on discrete sampling according to an embodiment of the present invention.

[0081] Dynamic object pose initialization plays a vital role in accurately locating the position of dynamic objects in the global map when they are first observed. Most existing methods estimate the initial pose of dynamic objects through a network. However, learning-based methods are difficult to generalize between different scenarios and difficult to adapt to complex autonomous driving scenarios. Therefore, the present invention proposes a sampling-based motion initialization method, which accurately estimates the 6DoF (six degrees of freedom) pose when a dynamic object is first observed. The process of dynamic object initialization is as follows: Figure 4As shown. Since almost all dynamic objects in autonomous driving scenarios are located on the ground, the present invention first sets the pitch angle (pitch) and roll angle (roll) to zero, and then calculates the optimal yaw angle (yaw) by sampling. The present invention first uses the RANSAC algorithm (Random Sample Consensus) to fit the ground and outputs the fitted ground equation, as shown in formula (1):

[0082] n1x+n2y+n3z+d=0(1),

[0083] Wherein, n=[n1, n2, n3] is the normal vector of the fitted plane, and d is the height of the plane. According to the equation of the fitted ground, the current camera self-motion posture is adjusted to align the ground horizontally with the world coordinate system and the height is set to 0. Subsequently, the present invention projects the mask of the dynamic object obtained in step 1 into the world coordinate system, and performs discrete sampling of the yaw angle from 0 to 180 degrees on the two-dimensional pixel plane. For each sampling result, a minimum area bounding box of the two-dimensional projection area is calculated. Then, the bounding box is used as a top view of the 3D bounding box of the moving object, thereby estimating the parameters of the 3D bounding box [t x ,t y ,yaw,length,width]. Then, using the constraint that all 3D points in the yaw direction need to be within the 3D bounding box, the remaining two parameters [t z ,height] can also be calculated.,Finally, the initial pose of the dynamic object is generated.

[0084] According to an embodiment of the present invention, the above-mentioned method of using the dynamic contour key point tracking method to extract semantic feature points on the contour of the dynamic object of the initial frame image to obtain the dynamic contour key points of the dynamic object of the initial frame image, and using the pixel tracking method to perform pixel-level tracking on the dynamic contour key points to obtain the tracking result of the dynamic contour key points of the non-initial frame image in the current video clip includes: using the dynamic contour key point tracking method to extract semantic feature points on the contour of the dynamic object of the initial frame image in the current video clip to obtain the dynamic contour key points of the dynamic object of the initial frame image; using the pixel tracking method to perform pixel-level tracking on each non-initial frame image to obtain the matching tracking result of the dynamic contour key points of each non-initial frame image; according to the preset key point screening conditions, using the random sampling consensus algorithm to screen the dynamic contour key points of the dynamic object of each non-initial frame image to obtain the screened dynamic contour key points of each non-initial frame image.

[0085] According to an embodiment of the present invention, the above-mentioned tracking results of dynamic contour key points based on non-initial frame images, using the perspective-3D algorithm to generate a preliminary estimate of the position and posture of the dynamic object in the current video clip includes: based on the correspondence between the dynamic contour key points between adjacent video frames in the current video clip, using the PnP algorithm to generate a preliminary estimate of the position and posture of the dynamic object in the current video clip.

[0086] In generating a preliminary estimate of the pose of a dynamic object in a current video clip, a StarMap neural network is used to obtain the dynamic contour key points of the dynamic object in the initial frame image in the current video clip, and a pixel tracking method is used to track the dynamic contour key points of the initial frame image at the pixel level to obtain the dynamic contour key points of the dynamic object in the next frame image adjacent to the initial frame, and a RANSAC (random sampling consensus algorithm) algorithm is used to filter the dynamic contour key points of the current non-initial frame image to eliminate the external points of the current non-initial frame image to obtain the filtered dynamic contour key points; based on the correspondence represented by the filtered dynamic contour key points, a PnP algorithm is used to generate a preliminary estimate of the pose of the dynamic object in the current non-initial frame; the above-mentioned dynamic contour key point tracking operation, pixel-level tracking operation, key point filtering and external point deletion operation and preliminary estimate of the pose of the dynamic object of the video frame are repeated for each non-initial frame image of the current video clip until a preliminary estimate of the pose of the dynamic object in each frame image of the current video clip is obtained.

[0087] The above embodiment involves operations S130 to S140 . The object pose estimation process based on pixel-level tracking of dynamic contour feature points involved in operations S130 to S140 will be further described in detail below through specific implementation methods.

[0088] First, dynamic contour key point tracking enhances the key point representation of dynamic objects by introducing contour and semantic information, and is combined with pixel tracking methods to establish long-term robust spatiotemporal association of dynamic objects. Subsequently, based on the results of dynamic contour point tracking, the present invention uses a simple but effective PnP algorithm (i.e., a three-dimensional-perspective algorithm) to obtain a preliminary estimate of the dynamic object's pose. The PnP algorithm is a method for estimating 6DoF motion based on matching points, and is mainly used to solve the target motion when the 2D-3D point correspondences of consecutive frames are known. Among them, P3P only requires three 2D-3D matching point pairs to estimate the motion pose. Thanks to the high precision of dynamic object contour feature point tracking, the present invention requires at least four 2D-3D correspondences to estimate the motion of dynamic objects, where an additional pair of matching points is used to verify the correctness of the estimated pose. The present invention evenly extracts ten key points on each detected dynamic object contour, and uses RANSAC to eliminate outliers and obtain a preliminary estimate of the moving object's pose. In addition, the preliminary estimated target object pose is optimized by minimizing the reprojection error cost function. After applying the PnP algorithm to generate a preliminary motion estimate, key frame I k The reprojection error of a contour key point in can be expressed by formula (2):

[0089]

[0090] in, The present invention uses Lie algebra to Parameterize and combine the dynamic contour key points and dynamic object motion estimation in the local map. The cost function at this time is shown in formula (3):

[0091]

[0092] Where m is the number of point pairs matching dynamic contour feature points. Due to occlusion, the number of contour feature points can gradually decrease during camera motion, leading to pixel tracking failures. To ensure the robustness of the P3P Solver, when the number of currently tracked dynamic contour keypoints decreases, the algorithm provided by the present invention detects new dynamic contour keypoints in the current frame and sets the current frame as a map keyframe. The new keyframe is then added to the map and the feature points are updated.

[0093] According to an embodiment of the present invention, the above-mentioned dynamic adjustment and optimization of the factor graph to generate the dynamic pose of the dynamic object in the current video includes: based on the tracking results of the dynamic contour key points of each non-initial frame image, detecting the change in the number of dynamic contour key points of the current non-initial frame image to obtain the detection result; when the detection result meets the predefined change condition, obtaining new dynamic contour key points from the current non-initial frame image, and setting the current non-initial frame image as the map key frame image of the current video clip; adding the map key frame image to the map of the current video clip and updating the dynamic contour key points of the map key frame image; based on the reprojection error of the map key frame image, using Lie algebra to parameterize the rigid body transformation of the dynamic object in the map key frame image to obtain the reprojection error cost function; fixing the estimation result of the camera self-motion pose of the current video clip, using a heuristic algorithm to adjust the weights in the factor graph, and optimizing the preliminary estimation of the pose of the dynamic object of each non-initial frame image by minimizing the reprojection error cost function to generate the dynamic pose of the dynamic object in the current video.

[0094] The above embodiment involves operation S150 . The following further describes in detail the bundle adjustment optimization framework based on dynamic contour key point tracking involved in operation S150 through specific implementations and in conjunction with the accompanying drawings.

[0095] Figure 5 is a schematic diagram of a bundle adjustment optimization framework based on dynamic contour key point tracking according to an embodiment of the present invention.

[0096] The backend of the SLAM problem is usually modeled as a factor graph optimization problem to optimize and improve the accuracy of camera and object motion estimation. Based on the contour key point tracking method, this paper designs a bundle adjustment optimization framework based on contour point tracking to further improve the accuracy of dynamic object motion estimation within the local map. Figure 5 As shown in the figure, in order to better express dynamic scenes, the present invention first fixes the camera pose and then constructs a sliding window internal factor graph to optimize the motion estimation of dynamic objects. In addition, this framework calls the sampling-based motion initialization method again every 5 key frames to correct the pose of the current frame so that it remains horizontal to the world coordinate system and has a height of zero. Finally, all dynamic contour key points, camera self-motion, dynamic object motion and sampling-based dynamic object motion are constructed into Figure 5 The factor graph is shown, and the motion of the moving object is optimized using the factor graph. Since the dynamic object motion obtained by the heuristic method (dynamic object motion initialization method based on sampling) usually has higher accuracy, the invention increases the weight of the object motion obtained by the sampling method in the factor graph (i.e. Figure 5 ) to obtain better optimization results.

[0097] Alternatively, the SLAM system can use the mainstream open-source feature-based visual SLAM system ORB-SLAM2. Its input can be monocular, binocular, or RGB-D video streams. For monocular input, a monocular depth prediction network is used to generate a depth map for each frame. The SLAM system's front-end odometry then estimates the robot's self-motion based on the depth map and video stream. Simultaneously, a local map is constructed from multiple frames, and the robot's self-motion within the local map is optimized.

[0098] Optionally, the object contour key points P are combined in the back-end bundle adjustment (BA) framework of the visual SLAM system o , static feature point P w , camera pose T ego And the pose T of the moving object calculated by PnP Solver dyn Joint optimization. This framework constructs a factor graph optimization model based on the constraint that the three-dimensional coordinates of the contour feature points in the object coordinate system remain unchanged. ego The remaining three variables are jointly optimized, and the sparsity of the Hessian matrix is ​​used to accelerate the calculation and reduce the computational complexity.

[0099] The present invention integrates an odometer front-end with an instance segmentation network to extract semantic information from the video stream while ensuring the real-time performance of the system, and screens out absolute dynamic objects in the scene based on heuristic settings. The SLAM front-end performs high-precision robot self-motion pose estimation based on the refined static background. The sampling-based dynamic object pose estimation method ensures the robust pose initialization of the dynamic object in the world coordinate system when it is first observed. The pixel tracking method based on contour key points establishes long-term robust spatiotemporal association of dynamic object contour points in the 2D pixel coordinate system. The PnP Solver based on contour key point tracking provides a preliminary dynamic object pose estimation. The subsequent bundle adjustment optimization framework based on contour key point tracking combines the object contour key points, the three-dimensional coordinates of the spatial point, the camera pose, and the PnP Solver motion object pose estimation to optimize the dynamic object pose, obtain accurate object motion, and thus generate an accurate dynamic object motion trajectory. This solution constructs long-term robust spatiotemporal association of dynamic object contour key points and performs high-precision dynamic object pose estimation.

[0100] Figure 6 A block diagram of an electronic device suitable for implementing a method for estimating the pose of a dynamic object based on dynamic contour key point tracking according to an embodiment of the present invention is schematically shown.

[0101] like Figure 6As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0102] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes the programs in ROM 602 and / or RAM 603 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute the programs stored in one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0103] According to an embodiment of the present invention, electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to bus 604. Electronic device 600 may further include one or more of the following components connected to I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. Communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in drive 610 as needed, so that computer programs read from the removable media can be installed into storage section 608 as needed.

[0104] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0105] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0107] The above specific embodiments further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for estimating the pose of a dynamic object based on dynamic contour key point tracking, characterized in that: include: Utilize the instance segmentation network to perform instance segmentation on each frame of the video data to obtain the instance segmentation results of each frame, extract dynamic objects from the instance segmentation results of each frame, and filter the instance segmentation results of each frame according to predefined filtering conditions to obtain the mask of the dynamic objects in each frame of the video data; Obtaining a fitting ground equation of an initial frame image in a current video segment using a random sampling consensus algorithm, and generating an initial pose of a dynamic object in the initial frame image based on the fitting ground equation of the initial frame image and a mask of the dynamic object in the initial frame image, wherein the initial frame image represents a frame image in which the dynamic object is first observed in the current video segment, and the current video segment represents continuous video frames intercepted from the video data according to predefined conditions; Extracting semantic feature points on the contour of the dynamic object in the initial frame image using a dynamic contour key point tracking method to obtain dynamic contour key points of the dynamic object in the initial frame image, and tracking the dynamic contour key points at the pixel level using a pixel tracking method to obtain tracking results of the dynamic contour key points of the non-initial frame image in the current video clip; Based on the tracking results of the dynamic contour key points of the non-initial frame image, a perspective-3D algorithm is used to generate a preliminary estimate of the position and posture of the dynamic object in the current video clip, and an image positioning and mapping technology is used to obtain an estimate of the self-motion position and posture of the camera in the current video clip; A factor graph of the dynamic object in the current video clip is constructed using the dynamic contour key points of the current video clip, the initial posture of the dynamic object, the preliminary estimation of the posture of the dynamic object, and the estimation result of the camera's self-motion posture. The dynamic posture of the dynamic object in the current video is generated by dynamically adjusting and optimizing the factor graph.

2. The method according to claim 1, characterized in that Also includes: The motion pose of the dynamic object in each video clip in the video data is optimized by using a bundle adjustment optimization framework based on dynamic contour key point tracking to obtain the motion trajectory of the dynamic object in the video data.

3. The method according to claim 1, characterized in that Performing instance segmentation on each frame of the video data using an instance segmentation network to obtain an instance segmentation result for each frame of the video data, extracting dynamic objects from the instance segmentation result for each frame of the video data, and filtering the instance segmentation result for each frame of the video data according to a predefined filtering condition to obtain a mask of the dynamic objects for each frame of the video data. Acquiring the video data through a video acquisition device, wherein the video acquisition device includes a monocular camera, a binocular camera, and a depth camera; Performing instance segmentation on each frame of the video data using the instance segmentation network, and extracting dynamic objects from the instance segmentation results of each frame of the image to obtain the dynamic objects in each frame of the image, wherein the instance segmentation network includes a SOLO instance segmentation network; Determining a predefined screening condition based on an overlap ratio of object masks between adjacent frame images in the video data and a ratio of object pixels to full image pixels in each frame image; According to the predefined filtering condition, static objects and objects smaller than a preset size are filtered out from the instance segmentation results of each frame of image to obtain a mask of the dynamic objects in each frame of image.

4. The method according to claim 3, characterized in that Acquiring the video data through a video acquisition device includes: The video data collected by the monocular camera is preprocessed through a monocular depth estimation network to obtain a depth map of each frame image in the video data.

5. The method according to claim 1, wherein Obtaining a fitting ground equation of an initial frame image in a current video clip using a random sampling consensus algorithm, and generating an initial pose of a dynamic object in the initial frame image based on the fitting ground equation of the initial frame image and a mask of the dynamic object in the initial frame image includes: Setting the pitch angle and roll angle of the moving object in the initial frame image according to preset values, and fitting the ground of the initial frame image using a random sampling consensus algorithm to obtain a fitting ground equation of the initial frame image; Adjusting the camera self-motion pose of the current video clip according to the fitted ground equation of the initial frame image, and setting the ground of the initial frame image based on the adjustment result and the world coordinate system; The mask of the dynamic object in the initial frame image is projected into the world coordinate system, and the yaw angle of the dynamic object is discretely sampled within a preset angle range on a two-dimensional pixel plane using the projection result to obtain a discrete sampling result. Calculating a minimum area bounding box of the discrete sampling result in a two-dimensional projection area, and using the minimum area bounding box of the discrete sampling result as a top view of the 3D bounding box of the dynamic object in the initial frame image to estimate the yaw angle, X-axis initial point, Y-axis initial point, length, and width of the 3D bounding box of the dynamic object in the initial frame image; Calculating a Z-axis initial point and a height of the 3D bounding box of the dynamic object based on a constraint condition between the yaw angle of the dynamic object in the initial frame image and the 3D bounding box of the dynamic object in the initial frame image to obtain all parameter values ​​of the 3D bounding box of the dynamic object; An initial posture of the dynamic object in the initial frame image is generated based on all parameter values ​​of the 3D bounding box of the dynamic object in the initial frame image, wherein the initial posture of the dynamic object is a posture with six degrees of freedom.

6. The method according to claim 1, wherein Extracting semantic feature points on the contour of the dynamic object in the initial frame image using a dynamic contour key point tracking method to obtain dynamic contour key points of the dynamic object in the initial frame image, and tracking the dynamic contour key points at the pixel level using a pixel tracking method to obtain tracking results of the dynamic contour key points of the non-initial frame image in the current video clip, including: Extracting semantic feature points on the contour of the dynamic object in the initial frame image in the current video clip using the dynamic contour key point tracking method to obtain dynamic contour key points of the dynamic object in the initial frame image; Performing pixel-level tracking on each of the non-initial frame images using the pixel tracking method to obtain matching tracking results of dynamic contour key points of each of the non-initial frame images; According to the preset key point screening conditions, the dynamic contour key points of the dynamic object in each non-initial frame image are screened using the random sampling consensus algorithm to obtain the screened dynamic contour key points of each non-initial frame image.

7. The method according to claim 1, characterized in that Generating the dynamic pose of the dynamic object in the current video by dynamically adjusting and optimizing the factor graph includes: Based on the tracking results of the dynamic contour key points of each non-initial frame image, detecting the change in the number of the dynamic contour key points of the current non-initial frame image to obtain a detection result; When the detection result satisfies a predefined change condition, obtaining new dynamic contour key points from the current non-initial frame image, and setting the current non-initial frame image as a map key frame image of the current video segment; Adding the map key frame image to the map of the current video segment and updating the dynamic contour key points of the map key frame image; Based on the reprojection error of the map key frame image, the rigid body transformation of the dynamic object in the map key frame image is parameterized using Lie algebra to obtain a reprojection error cost function; Fix the estimated result of the camera self-motion pose of the current video clip, use a heuristic algorithm to adjust the weights in the factor graph, and optimize the preliminary estimate of the pose of the dynamic object in each non-initial frame image by minimizing the reprojection error cost function to generate the dynamic pose of the dynamic object in the current video.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 7.

Images