Dense Reconstruction Method, System and Storage Medium for Static Environment and Dynamic Objects

Abstract

The present invention discloses a method, system and medium for dense reconstruction of a static environment and dynamic objects. The method includes: performing instance segmentation on the input image; extracting and matching first feature points on the static image to calculate the initial pose of the camera; extracting and tracking second feature points on the instance image to calculate the initial relative pose of the dynamic object; performing tightly coupled non-linear optimization on the constraint relationships of the pose of the camera, the relative pose of the dynamic object, the coordinates of the first feature points and the coordinates of the second feature points; obtaining a depth map through depth estimation of the input image and performing preprocessing; establishing a dense point cloud map for the static environment; screening the object point cloud fragments, and transforming the retained point cloud fragments to the reference fragment for fusion, so as to densely reconstruct the dynamic object. By inputting binocular images, the present invention can output a dense reconstruction point cloud map of a static environment and dynamic objects, and the whole process is automatically processed, having the characteristics of high precision, less ghosting and noise, and good robustness.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and 3D reconstruction, and particularly relates to a method, system and storage medium for dense reconstruction of a static environment and dynamic objects. Background Art

[0002] 3D reconstruction is an important field in computer vision, aiming to obtain observation information through sensors and reconstruct the surface contour information of objects in the real environment after processing and analyzing the observation information. 3D reconstruction has strong application prospects and is widely used in fields such as virtual reality, digital twin, autonomous driving, and 3D printing.

[0003] The initial 3D reconstruction technologies were mainly based on active sensors with ranging functions such as laser sensors and radars. However, their equipment costs are expensive, the usage methods are cumbersome, and the provided observation information is also relatively small, which limits their applications. The vision-based 3D reconstruction technology can restore the 3D model of a scene or an object by using a single or multiple images. This process can actually be regarded as the reverse process of photography. The vision-based 3D reconstruction technology has the advantages of low cost, convenient measurement, rich information, and fast processing speed, and has received the key attention of researchers and has developed vigorously.

[0004] Current vision-based 3D reconstruction algorithms mainly restore 3D scene information by estimating the camera pose and pixel depth, such as the 3D scene reconstruction method based on GPU acceleration disclosed in the Chinese invention patent with the publication number CN103021017B. Such scene reconstruction methods have been fully studied. However, they often assume that the scene observed by the camera is completely static, that is, the so-called rigid body assumption. However, in reality, it is not easy to meet this condition, especially in outdoor scenes. If 3D reconstruction is still carried out using the rigid body assumption in a highly dynamic scene, the reconstructed model will inevitably be full of noise and trailing caused by dynamic objects, which destroys the static model. Furthermore, in a dynamic scene, in addition to reconstructing the static part, it is also necessary to model the dynamic objects therein, which can be applied to uses such as obstacle avoidance in autonomous driving. Modeling of dynamic objects requires not only separating them from the static environment but also tracking their movements, which poses higher requirements for vision-based 3D reconstruction algorithms. Summary of the Invention

[0005] The main purpose of the present invention is to overcome the deficiencies of the prior art and provide a method, system and storage medium for dense reconstruction of a static environment and dynamic objects. This method uses instance segmentation to separate the static environment and dynamic objects, estimates and optimizes the poses of the camera and dynamic objects respectively, and performs depth estimation and depth map preprocessing. Finally, dense reconstruction of the static environment and dynamic objects is achieved, which has the characteristics of high accuracy, less trailing and noise, and good robustness.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for dense reconstruction of a static environment and dynamic objects, comprising the following steps:

[0008] Perform instance segmentation on the input image to obtain a static image and an instance image;

[0009] Extract and match first feature points on the static image, and calculate the initial pose of the camera based on the first feature points;

[0010] Extract and track second feature points on the instance image, and calculate the initial relative pose of the dynamic object based on the second feature points;

[0011] Perform tightly coupled non-linear optimization on the constraint relationships of the camera pose, the relative pose of the dynamic object, the first feature point coordinates, and the second feature point coordinates. The tightly coupled non-linear optimization includes the camera pose constraint relationship and the relative pose constraint relationship of the dynamic object;

[0012] Obtain a depth map by depth estimation of the input image, and preprocess the depth map by using morphological methods and edge detection methods;

[0013] Back-project the static pixel points on the depth map into the camera coordinate system, and transform them into the world coordinate system by using the optimized camera pose, so as to establish a dense point cloud map for the static environment;

[0014] Transform the dynamic object pixel points on the depth map into the world coordinate system, obtain multiple point cloud segments of the same object, screen the point cloud segments, and finally transform the remaining point cloud segments into the reference segment for fusion by using the optimized relative pose of the dynamic object, so as to densely reconstruct the dynamic object.

[0015] As a preferred technical solution, the performing instance segmentation on the input image is specifically:

[0016] Adopt the BlendMask algorithm to perform instance segmentation on the input RGB image to obtain a background mask and an instance mask, and apply the background mask and the instance mask to the original RGB image to obtain a static image and an instance image, so as to separate each object from the background.

[0017] As a preferred technical solution, the extracting and matching first feature points on the static image and calculating the initial pose of the camera based on the first feature points is specifically:

[0018] Extract the first feature points on the front and rear static images respectively. The first feature points are ORB feature points. Calculate the descriptors of the first feature points and perform matching of the first feature points according to the Hamming distance of the descriptors. Calculate the coordinates in the world coordinate system based on the matched first feature points in this frame. It has the following projective transformation relationship:

[0019]

[0020] Among them, represents the observed coordinates corresponding in the pixel coordinates, and π(·) represents the projection process. represents the transformation matrix of the camera from the world coordinate system to the camera coordinate system in the k-th frame. According to the above relationship, use the EPnP algorithm to calculate the pose of the camera in the k-th frame.

[0021] As a preferred technical solution, extract and track the second feature points on the instance images, and calculate the initial relative pose of the dynamic object based on the second feature points. Specifically:

[0022] Extract the second feature points from the front and rear instance images. The second feature points are Shi-Tomasi feature points. Use the optical flow method to calculate the offset of the same second feature point between the two images to complete the tracking of the second feature points.

[0023] After obtaining the initial pose of the camera, calculate the coordinates of the second feature points on each instance image in the world coordinate system. The matching relationship of the second feature points between two instance images has been obtained. There is the following pose transformation relationship:

[0024]

[0025] Among them, and respectively represent the coordinates of the j-th second feature point in the instance image in the world coordinate system in the (k - 1)-th frame and the k-th frame. Since the dynamic object is in continuous motion, these two coordinates are different. is the pose transformation of the dynamic object in the world coordinate system from the (k - 1)-th frame to the k-th frame. Since and are both known, the ICP algorithm is used to solve the pose transformation.

[0026] As a preferred technical solution, the pose constraint relationship of the camera is expressed as follows:

[0027]

[0028] Among them, e k,iDenote the pose of the camera at the k-th frame and the 3D coordinates of the i-th first feature point in the k-th frame in the world coordinate system to form the residual, and both are optimization variables;

[0029] The relative pose constraint relationship of the dynamic object is expressed as follows:

[0030]

[0031] where e k,n,j denotes the j-th second feature point on the n-th object in the k-th frame and its corresponding second feature point in the (k - 1)-th frame, as well as the transformation matrix between the two to form the residual, and all three are optimization variables;

[0032] Therefore, the total tightly coupled loss function is expressed as follows:

[0033]

[0034] where K represents the number of key frames, N k denotes the number of dynamic objects in the k-th frame, I k denotes the number of static feature points in the k-th frame, J k,n denotes the number of dynamic feature points on the n-th dynamic object in the k-th frame, T wc , B and w m and w p are macro variable representations of the batch pose of the camera to be optimized, the batch pose of all dynamic objects, the world coordinate system coordinates of static feature points, and the world coordinate system coordinates of dynamic feature points respectively. Σ r and Σ o are the covariance matrices of the measurement noises of static feature points and dynamic feature points respectively.

[0035] As a preferred technical solution, the depth map is obtained through depth estimation, and morphological methods and edge detection are used to preprocess the depth map. Specifically:

[0036] Use the STTR algorithm to process the input binocular RGB image to solve for a smooth depth map; then, using the mask image, the depth map is also separated into a static depth map and an instance depth map. Since instance segmentation cannot classify each pixel completely accurately, before obtaining the static depth map, it is necessary to dilate the mask image to filter out the dynamic pixels; and before obtaining the instance depth map, the mask image is eroded to filter out the static pixels;

[0037] Perform Canny edge detection on the static depth map to extract the edges of static objects, convert the edges into a mask, and dilate the edge mask to cover more outlier pixels; when establishing the static point cloud map, the pixels in the edge mask will not be included in the point cloud calculation range.

[0038] As a preferred technical solution, the static pixel points on the depth map are back-projected into the camera coordinate system and transformed into the world coordinate system using the optimized camera pose, so as to establish a dense point cloud map for the static environment. Specifically:

[0039] For the pixels outside the edge mask on the static depth map, back-project them into the camera coordinate system and transform them into the world coordinate system using the optimized camera pose, that is:

[0040]

[0041] Among them, and respectively represent the homogeneous coordinates of the i-th feature point in the k-th frame of the static image in the pixel coordinate system and the coordinates in the world coordinate system, Z represents the depth value of the feature point in the camera coordinate system, and K represents the internal parameter matrix under the camera pinhole model; after coordinate transformation, the point cloud color can be obtained from the corresponding pixel on the RGB image. By performing the above transformation on the available pixels on the static depth map in all key frames and fusing the obtained point clouds, a dense static point cloud map is obtained.

[0042] As a preferred technical solution, the dynamic object pixel points on the depth map are transformed into the world coordinate system to obtain multiple point cloud segments of the same object, the point cloud segments are screened, and finally the remaining point cloud segments that meet the threshold are transformed into the reference segment using the optimized relative pose of the dynamic object for fusion, so as to densely reconstruct the dynamic object. Specifically:

[0043] Transform the dynamic pixel points of the dynamic object on the depth map corresponding to each key frame into the world coordinate system, so as to obtain multiple point cloud segments, select the point cloud segment with the most points as the reference segment, and transform the segments that meet the threshold in the remaining point cloud segments into the pose of the reference segment, that is

[0044]

[0045]

[0046] Among them, represents the set of dynamic object point cloud segments, N k represents the number of points in the k-th point cloud segment, and λ represents the threshold for filtering the point cloud segments. Through the above operations, some point cloud segments are fused to obtain the point cloud map of the dynamic object.

[0047] In a second aspect, the present invention provides a dense reconstruction system for a static environment and dynamic objects, characterized in that it applies the dense reconstruction method for the static environment and dynamic objects, including an image acquisition module, a first pose calculation module, a second pose calculation module, an optimization module, a depth map processing module, a dense point cloud map construction module, and a dense reconstruction module;

[0048] The image acquisition module is used to perform instance segmentation on the input image to obtain a static image and an instance image;

[0049] The first pose calculation module is used to extract and match first feature points on the static image, and calculate the initial pose of the camera based on the first feature points;

[0050] The second pose calculation module is used to extract and track second feature points on the instance image, and calculate the initial relative pose of the dynamic object based on the second feature points;

[0051] The optimization module is used to perform tightly coupled non-linear optimization on the constraint relationships of the initial pose of the camera, the initial relative pose of the dynamic object, the coordinates of the first feature points, and the coordinates of the second feature points. The tightly coupled non-linear optimization includes the pose constraint relationship of the camera and the relative pose constraint relationship of the dynamic object;

[0052] The depth map processing module is used to obtain a depth map by depth estimation of the input image, and preprocess the depth map by using morphological methods and edge detection methods;

[0053] The dense point cloud map construction module is used to back-project the static pixel points on the depth map into the camera coordinate system, and transform them into the world coordinate system by using the optimized camera pose, so as to establish a dense point cloud map for the static environment;

[0054] The dense reconstruction module is used to transform the dynamic object pixel points on the depth map into the world coordinate system, obtain multiple point cloud segments of the same object, screen the point cloud segments, and finally use the optimized relative pose of the dynamic object to transform the remaining point cloud segments into the reference segment for fusion, so as to densely reconstruct the dynamic object.

[0055] In a third aspect, the present invention provides a computer-readable storage medium storing a program, which when executed by a processor, implements the dense reconstruction method for the static environment and dynamic objects.

[0056] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0057] 1. The present invention performs instance segmentation on the input image to separate the static environment and dynamic objects, estimates the poses of the camera and dynamic objects respectively, and performs dense reconstruction, avoiding the adverse effects of dynamic pixels on pose estimation and the ghosting problem during reconstruction.

[0058] 2. The present invention performs tightly coupled non - linear optimization on the camera pose, dynamic object pose, and feature point coordinates, which helps to reduce the cumulative error of the system, ensure the smoothness of the motion trajectory, and the fusion effect of dense reconstruction.

[0059] 3. The present invention performs canny edge detection on the depth map, improves the inaccurate depth estimation of object contours, and helps to construct an accurate static point cloud map. At the same time, a point cloud fragment fusion strategy based on screening is adopted for the construction of the point cloud map of dynamic objects, improving the reconstruction effect of dynamic objects. Description of the Drawings

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0061] Figure 1 It is a flowchart of the dense reconstruction method for the static environment and dynamic objects in the embodiment of the present invention;

[0062] Figure 2 It is a binocular camera imaging model in the embodiment of the present invention.

[0063] Figure 3 It is an effect diagram of the dense reconstruction of the static environment in the embodiment of the present invention.

[0064] Figure 4 It is an effect diagram of the dense reconstruction of a certain dynamic vehicle in the field of view in the embodiment of the present invention.

[0065] Figure 5 It is a block diagram of the dense reconstruction system for the static environment and dynamic objects in the embodiment of the present invention.

[0066] Figure 6 It is a structural diagram of the computer - readable storage medium in the embodiment of the present invention. Detailed Embodiments

[0067] To enable those skilled in the art to better understand the solution of this application, the technical solution in the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of this application.

[0068] In this application, the mention of "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described in this application can be combined with other embodiments.

[0069] As Figure 1 shown, a method for dense reconstruction of a static environment and dynamic objects in this embodiment includes the following steps:

[0070] S1. Perform instance segmentation on the input image to obtain a static image and instance images.

[0071] Specifically, in this embodiment, the BlendMask algorithm is used to perform instance segmentation on the left-eye image in the input binocular RGB image, and a mask image can be obtained. This mask image is a grayscale image. Among them, the pixel value of the static environment is set to 0, and the instance pixels are set to different grayscale values according to different objects. Therefore, a static mask image and several instance images can be obtained through this mask image. If these two mask images are respectively applied to the original RGB image, a static image and instance images can be obtained, thereby separating each object from the background.

[0072] S2. Extract and match the first feature points (ORB feature points) on the static image, and calculate the initial pose of the camera. In this embodiment, the first feature points are the ORB feature points.

[0073] In this embodiment, the pose calculation of the camera is based on the ORB feature point method, which has a high matching accuracy, is less affected by light, and is relatively robust.

[0074] Further, step S2 is specifically as follows:

[0075] S21. It is necessary to extract FAST corner points on the static images of two consecutive frames respectively.

[0076] Exemplarily, during this extraction process, each pixel of the static image is traversed, and 16 pixels on a circle with a radius of 3 centered on this pixel are selected. It is detected whether there are 12 consecutive pixels greater than 120% of the gray value of this pixel or less than 80% of the gray value of this pixel. If this condition is met, it indicates that the change in the local gray value of this pixel point is very obvious, and it is classified as a FAST corner point.

[0077] S22. After extracting the feature points, it is necessary to calculate their BRIEF descriptors.

[0078] Exemplarily, when calculating the BRIEF descriptor, 128 pairs of pixel pairs near the feature points are randomly extracted, and their size relationships are compared, thereby generating a 128-dimensional binary description. In addition, this embodiment also calculates the direction of the feature points. Combining the feature point direction and the descriptor will make the descriptor have good rotational invariance and will be more robust when performing feature matching.

[0079] S23. After completing the extraction of ORB feature points from the front and rear two frames of static images, feature point matching will be performed according to the Hamming distance of the descriptors, that is, the number of different bits of the 128-dimensional binary numbers of the two feature point descriptors is counted. The fewer the number, the smaller the Hamming distance, and the greater the similarity degree of the two feature points.

[0080] S24. The coordinates in the world coordinate system can be calculated for the matched feature points in this frame It has the following projection transformation relationship:

[0081]

[0082] Among them, represents the observed coordinate corresponding in the pixel coordinate, and π(·) represents the projection process, represents the transformation matrix of the camera from the world coordinate system to the camera coordinate system in the k-th frame. According to the above relationship, the pose of the camera in the k-th frame can be calculated by means of the EPnP algorithm

[0083] S3. Extract and track the second feature points on the instance image and calculate the initial relative pose of the dynamic object. In this embodiment, the second feature points are Shi-Tomasi feature points.

[0084] Further, step S3 is specifically as follows:

[0085] S31. Extract Shi-Tomasi feature points from the previous frame of instance image and perform stereo matching using the optical flow method. According to the binocular principle, the depth of the feature points can be calculated, as Figure 2 shows the imaging model of the binocular camera. Among them, O Land O R respectively represent the centers of the left and right apertures, b represents the baseline length of the binocular camera, f represents the focal length of the camera, P represents a certain 3D point in the environment, and Z represents the depth of this point in the camera coordinate system. And u L and u R respectively represent the coordinates of this point in the left and right pixel coordinate systems of the camera. Then we have:

[0086]

[0087] After a little rearrangement, we have

[0088]

[0089] Among them, d = u L -u R , which is called parallax. Thus, the depth of the pixel point can be obtained according to the parallax of the pixel point in the left and right images, the baseline length, and the camera focal length.

[0090] S32. Track the feature points with successfully solved depth values to the next frame using the optical flow method. In step S2, the initial pose of the camera corresponding to this frame has been obtained, and the coordinates of the feature points on each frame of the instance image in the world coordinate system can be calculated according to this initial pose. Since the matching relationship of the feature points between two frames of instance images has been obtained, they have the following pose transformation relationship:

[0091]

[0092] Among them, and respectively represent the coordinates of the j-th feature point in the instance image in the world coordinate system in the (k - 1)-th frame and the k-th frame. Since the dynamic object is in continuous motion, these two coordinates are different. is the pose transformation of the dynamic object in the world coordinate system from the (k - 1)-th frame to the k-th frame. Since and are both known, the ICP algorithm can be used to solve the pose transformation

[0093] S4. Perform a tightly coupled non-linear optimization on the constraint relationships of the camera pose, the relative pose of the dynamic object, the first feature point coordinates, and the second feature point coordinates.

[0094] It can be understood that during the long-term movement of the camera, the pose estimation of the camera and the dynamic object will continuously accumulate errors, making the estimation results untrustworthy. Therefore, a tightly coupled method is used to perform non-linear optimization on the constraint relationships of the camera pose, the relative pose of the dynamic object, and the feature point coordinates.

[0095] Considering the time consumption of the optimization system, this embodiment does not perform tightly coupled nonlinear optimization on all frames, but only on key frames that meet the following conditions:

[0096] ① The number of inliers in the frame must exceed the set minimum number, and the overlap with the previous keyframe should not be too large to avoid introducing too much redundant information;

[0097] ② The frame should differ from the previous key frame by at least a minimum number of common frames to avoid too dense key frame insertion, which would increase the processing burden of tight coupling optimization;

[0098] ③ The frame differs from the previous key frame by more than a certain maximum number of ordinary frames. At this time, a key frame should be inserted to avoid tracking failure;

[0099] ④ When a new dynamic object is created, keyframes should be inserted to help improve the positioning accuracy of the dynamic object.

[0100] Furthermore, the tightly coupled nonlinear optimization includes the pose constraint relationship of the camera and the relative pose constraint relationship of the dynamic object, and the specific steps are as follows:

[0101] S41. One of the optimization factors in the tightly coupled optimization is the pose constraint of the camera, which can be expressed as:

[0102]

[0103] Among them, e k,i Represents the position of the camera at the kth frame and the 3D coordinates of the i-th feature point in the k-th frame in the world coordinate system The residuals are composed of, and both are optimization variables. π(·) represents the projection process. The result is recorded as Then the projection process can be expressed as:

[0104]

[0105] Among them, (uv 1) T express The homogeneous coordinates in the pixel coordinate system after projection, and K represents the intrinsic parameter matrix of the camera, which mainly includes f x 、f y 、c x 、c y Four parameters, the intrinsic parameter matrix can be obtained through calibration.

[0106] S42. Another optimization factor in tight coupling optimization is the relative position constraint relationship of dynamic objects, which can be expressed as:

[0107]

[0108] Among them, e k,n,j represents the j-th feature point on the n-th object in the k-th category and its corresponding feature point in the (k - 1)-th frame, as well as the transformation matrix between the two. The residuals formed by them, and all three are optimization variables.

[0109] S43. Therefore, the total tightly-coupled loss function of the system can be written as:

[0110]

[0111] Among them, K represents the number of key frames, N k represents the number of dynamic objects in the k-th frame, I k represents the number of static feature points in the k-th frame, J k,n represents the number of dynamic feature points on the n-th dynamic object in the k-th frame, T wc , B, and w m, and w p are macro variable representations of the batch pose of the camera to be optimized, the batch poses of all dynamic objects, the world coordinate system coordinates of static feature points, and the world coordinate system coordinates of dynamic feature points respectively. Σ r and Σ o are the covariance matrices of the measurement noises of static feature points and dynamic feature points respectively.

[0112] In another embodiment of the application, the solution to the above least squares problem can adopt the Levenberg-Marquardt algorithm. After multiple iterative optimizations, relatively accurate values of the optimization variables can be obtained.

[0113] S5. Obtain a depth map through depth estimation, and preprocess the depth map using morphological methods and edge detection.

[0114] Furthermore, step S5 is specifically as follows:

[0115] S51. Process the input binocular RGB image using the STTR algorithm, and a smooth depth map can be solved.

[0116] S52. Using the mask image, the depth map can also be separated into a static depth map and an instance depth map. Since instance segmentation cannot classify each pixel completely accurately, before obtaining the static depth map, the mask image needs to be dilated to filter out the dynamic pixels; while before obtaining the instance depth map, the mask image needs to be eroded to filter out the static pixels.

[0117] S53. In the static depth map, the depth estimation at the contour of static objects often has low accuracy. Therefore, after the point cloud map is established, there are often many outlier points attached to the edges of static objects. To solve this problem, the static depth map can be subjected to Canny edge detection to extract the edges of static objects and convert the edges into a mask. To cover more outlier pixels, the edge mask can be dilated. When establishing the static point cloud map, the pixels in the edge mask will not be included in the scope of point cloud calculation.

[0118] S6. Back-project the static pixel points on the depth map into the camera coordinate system and use the optimized camera pose to transform them into the world coordinate system, so as to establish a dense point cloud map for the static environment.

[0119] Further, in step S6, for the pixels outside the edge mask on the static image, they can be back-projected into the camera coordinate system and transformed into the world coordinate system using the optimized camera pose, that is:

[0120]

[0121] Among them, and respectively represent the homogeneous coordinates of the i-th feature point in the k-th frame of the static image in the pixel coordinate system and the coordinates in the world coordinate system, Z represents the depth value of the feature point in the camera coordinate system, and K represents the internal parameter matrix under the camera pinhole model. By performing the above conversions on the available pixels on the static images of all key frames, a dense static point cloud map can be obtained.

[0122] S7. Transform the dynamic object pixel points on the depth map into the world coordinate system, obtain multiple point cloud segments of the same object, screen the point cloud segments, and finally use the optimized relative pose of the dynamic object to transform the remaining point cloud segments to the reference segment for fusion, so as to densely reconstruct the dynamic object.

[0123] Further, in step S7, since the dynamic object is in continuous motion, a point cloud segment of the dynamic object can be obtained for each key frame. In this embodiment, the dynamic pixel points on the depth map corresponding to each key frame of the dynamic object are transformed into the world coordinate system, so as to obtain multiple point cloud segments. The segment with the most points is selected as the reference segment, and the segments that meet the threshold in the remaining segments are all transformed to the pose of the reference segment, that is

[0124]

[0125]

[0126] Among them, represents the set of point cloud segments of the dynamic object, N kLet \(n_k\) denote the number of points in the \(k\)-th point cloud segment, and \(\lambda\) denote the threshold for point cloud segment filtering. Through the above operations, some point cloud segments are fused to obtain the point cloud map of the dynamic object.

[0127] S8. Test this embodiment in an outdoor traffic scenario.

[0128] In this embodiment, a binocular color camera is adopted and installed on the roof of the test vehicle for testing on urban roads. Table 1 shows the relevant parameters of the camera.

[0129] Table 1

[0130]

[0131] During the test process, this embodiment will solve the pose of the camera in each frame. Moreover, multiple moving vehicles can be observed in the camera's field of view, and their relative poses in each frame will also be solved. With accurate pose estimation, as described in steps S5, S6, and S7, dense three-dimensional reconstruction of the static environment and dynamic vehicles can be performed. As Figure 3 shown, the dense reconstruction effect diagram of the static urban road environment. As Figure 4 shown, the dense reconstruction effect diagram of one of the dynamic vehicles in the field of view is presented. It can be seen that the reconstructed point cloud map has less trailing and noise, and the reconstruction accuracy is very high, well meeting the application requirements of three-dimensional reconstruction.

[0132] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously.

[0133] Based on the same idea as the dense reconstruction method of the static environment and dynamic objects in the above embodiments, the present invention also provides a dense reconstruction system for the static environment and dynamic objects, which can be used to execute the above-mentioned dense reconstruction method of the static environment and dynamic objects. For the sake of convenience of description, in the structural schematic diagram of the dense reconstruction system embodiment of the static environment and dynamic objects, only the parts related to the embodiments of the present invention are shown. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than those illustrated, or combine certain components, or have different component arrangements.

[0134] As Figure 5As shown, in another embodiment of the present application, a dense reconstruction system 100 for a static environment and dynamic objects is provided. The system includes an image acquisition module 101, a first pose calculation module 102, a second pose calculation module 103, an optimization module 104, a depth map processing module 105, a dense point cloud map construction module 106, and a dense reconstruction module 107;

[0135] The image acquisition module 101 is configured to perform instance segmentation on the input image to obtain a static image and an instance image;

[0136] The first pose calculation module 102 is configured to extract and match first feature points on the static image, and calculate the initial pose of the camera based on the first feature points;

[0137] The second pose calculation module 103 is configured to extract and track second feature points on the instance image, and calculate the initial relative pose of the dynamic object based on the second feature points;

[0138] The optimization module 104 is configured to perform tightly coupled non-linear optimization on the constraint relationships of the camera pose, the relative pose of the dynamic object, the coordinates of the first feature points, and the coordinates of the second feature points. The tightly coupled non-linear optimization includes the constraint relationship of the camera pose and the constraint relationship of the relative pose of the dynamic object;

[0139] The depth map processing module 105 is configured to obtain a depth map by depth estimation of the input image, and preprocess the depth map by using morphological methods and edge detection methods;

[0140] The dense point cloud map construction module 106 is configured to back-project the static pixel points on the depth map into the camera coordinate system, and transform them into the world coordinate system by using the optimized camera pose, so as to establish a dense point cloud map for the static environment;

[0141] The dense reconstruction module 107 is configured to transform the dynamic object pixel points on the depth map into the world coordinate system, obtain multiple point cloud segments of the same object, screen the point cloud segments, and finally transform the remaining point cloud segments into the reference segment for fusion by using the optimized relative pose of the dynamic object, so as to densely reconstruct the dynamic object.

[0142] It should be noted that the dense reconstruction system for a static environment and dynamic objects of the present invention corresponds one-to-one with the dense reconstruction method for a static environment and dynamic objects of the present invention. The technical features and beneficial effects described in the embodiments of the above-mentioned dense reconstruction method for a static environment and dynamic objects are applicable to the embodiments of the dense reconstruction system for a static environment and dynamic objects. For specific content, reference can be made to the description in the method embodiments of the present invention, which will not be elaborated here. This is hereby declared.

[0143] In addition, in the implementation of the dense reconstruction system for the static environment and dynamic objects in the above embodiments, the logical division of each program module is only for illustration. In practical applications, according to needs, for example, considering the configuration requirements of the corresponding hardware or the convenience of software implementation, the above functions can be assigned to different program modules to complete, that is, the internal structure of the dense reconstruction system for the static environment and dynamic objects is divided into different program modules to complete all or part of the functions described above.

[0144] As Figure 6 shown, in one embodiment, a computer-readable storage medium 200 is provided, storing a program in a memory 201. When the program is executed by a processor 202, the dense reconstruction method for the static environment and dynamic objects as described above is implemented, specifically:

[0145] Perform instance segmentation on the input image to obtain a static image and an instance image;

[0146] Extract and match first feature points on the static image, and calculate the initial pose of the camera based on the first feature points;

[0147] Extract and track second feature points on the instance image, and calculate the initial relative pose of the dynamic object based on the second feature points;

[0148] Perform tightly coupled non-linear optimization on the constraint relationships of the camera pose, the relative pose of the dynamic object, the coordinates of the first feature points, and the coordinates of the second feature points. The tightly coupled non-linear optimization includes the constraint relationship of the camera pose and the constraint relationship of the relative pose of the dynamic object;

[0149] Obtain a depth map by depth estimation for the input image, and preprocess the depth map using morphological methods and edge detection methods;

[0150] Back-project the static pixel points on the depth map into the camera coordinate system and transform them into the world coordinate system using the optimized camera pose, thereby establishing a dense point cloud map for the static environment;

[0151] Transform the dynamic object pixel points on the depth map into the world coordinate system, obtain multiple point cloud segments of the same object, screen the point cloud segments, and finally transform the remaining point cloud segments to the reference segment for fusion using the optimized relative pose of the dynamic object, thereby densely reconstructing the dynamic object.

[0152] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0153] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0154] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention should be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A method for dense reconstruction of a static environment and dynamic objects, characterized in that, Including the following steps: Performing instance segmentation on the input image to obtain a static image and an instance image; Extracting and matching first feature points on the static image, and calculating the initial pose of the camera based on the first feature points; Extracting and tracking second feature points on the instance image, and calculating the initial relative pose of the dynamic object based on the second feature points; Performing tightly coupled non-linear optimization on the constraint relationships of the camera pose, the relative pose of the dynamic object, the coordinates of the first feature points, and the coordinates of the second feature points, where the tightly coupled non-linear optimization includes the constraint relationship of the camera pose and the constraint relationship of the relative pose of the dynamic object; Obtaining a depth map by depth estimation for the input image, and preprocessing the depth map using morphological methods and edge detection methods. Specifically: Processing the input binocular RGB image using the STTR algorithm to solve for a smooth depth map; then, using the mask image, the depth map is also separated into a static depth map and an instance depth map. Since instance segmentation cannot classify each pixel completely accurately, before obtaining the static depth map, it is necessary to perform dilation processing on the mask image to filter out the dynamic pixels; while before obtaining the instance depth map, erosion processing is performed on the mask image to filter out the static pixels; Performing canny edge detection on the static depth map to extract the edges of the static object, and converting the edges into a mask. To cover more outlier pixels, dilation processing is performed on the edge mask; when establishing the static point cloud map, the pixels in the edge mask will not be included in the point cloud calculation range; Back-projecting the static pixel points on the depth map into the camera coordinate system and converting them to the world coordinate system using the optimized camera pose, thereby establishing a dense point cloud map for the static environment; Transforming the dynamic object pixel points on the depth map into the world coordinate system to obtain multiple point cloud segments of the same object, screening the point cloud segments, and finally using the optimized relative pose of the dynamic object to transform the remaining retained point cloud segments to the reference segment for fusion, thereby densely reconstructing the dynamic object. Specifically: Transforming the dynamic pixel points on the depth map corresponding to each key frame of the dynamic object into the world coordinate system, thereby obtaining multiple point cloud segments, selecting the point cloud segment with the most points as the reference segment, and transforming the segments that meet the threshold in the remaining point cloud segments to the pose of the reference segment, that is Among them, represents a set of point cloud segments of dynamic objects, and N t represents the number of points in the t-th point cloud segment, and λ represents the threshold for filtering point cloud segments. Through the above operations, some point cloud segments are fused to obtain the point cloud map of the dynamic object.

2. The dense reconstruction method for static environment and dynamic objects according to claim 1, wherein The performing instance segmentation on the input image is specifically: Performing instance segmentation on the input RGB image using the BlendMask algorithm to obtain a background mask and an instance mask, and applying the background mask and the instance mask to the original RGB image to obtain a static image and an instance image, thereby separating each object from the background.

3. The dense reconstruction method for static environment and dynamic objects according to claim 2, characterized in that The extracting and matching first feature points on the static image and calculating the initial pose of the camera based on the first feature points is specifically: Extract the first feature points on the front and rear static images respectively, where the first feature points are ORB feature points, calculate the descriptors of the first feature points, match the first feature points according to the Hamming distance of the descriptors, and calculate the coordinates in the world coordinate system based on the matched first feature points in this frame It has the following projective transformation relationship: Among them, represents the observed coordinates corresponding in pixel coordinates, and π(·) represents the projection process. represents the transformation matrix of the camera from the world coordinate system to the camera coordinate system in the k-th frame. According to the above relationship, the pose of the camera in the k-th frame is calculated using the EPnP algorithm.

4. The dense reconstruction method for static environment and dynamic objects according to claim 3, characterized in that The extracting and tracking second feature points on the instance image and calculating the initial relative pose of the dynamic object based on the second feature points is specifically: Extract the second feature points from the front and rear two-frame instance images. The second feature points are Shi-Tomasi feature points. Use the optical flow method to calculate the offset of the same second feature point between two frames of images to complete the tracking of the second feature points. After obtaining the initial pose of the camera, calculate the coordinates of the second feature points on each frame of instance image in the world coordinate system. The matching relationship of the second feature points between two frames of instance images has been obtained. There is the following pose transformation relationship: Among them, and respectively represent the coordinates of the j-th second feature point in the instance image in the k-1-th frame and the k-th frame in the world coordinate system. Since the dynamic object is in continuous motion, these two coordinates are different. is the pose transformation of the dynamic object in the world coordinate system from the k-1-th frame to the k-th frame. Since and are both known, the ICP algorithm is used to solve the pose transformation.

5. The dense reconstruction method for a static environment and dynamic objects according to claim 4, wherein, The pose constraint relationship of the camera is expressed as follows: Among them, e k,i represents the pose of the camera at the k-th frame and the coordinates of the i-th first feature point in the k-th frame in the world coordinate system constitute the residual, and both are optimization variables; The relative pose constraint relationship of the dynamic object is expressed as follows: where, e k,n,j represents the j-th second feature point on the n-th object in the k-th group and its corresponding second feature point in the (k - 1)-th frame, as well as the transformation matrix between the two which constitutes the residual, and all three are optimization variables; Therefore, the total tightly coupled loss function is expressed as follows: Among them, K represents the number of key frames, N k represents the number of dynamic objects in the k-th frame, I k represents the number of static feature points in the k-th frame, J k,n represents the number of dynamic feature points on the n-th dynamic object in the k-th frame, T wc 、B and w m and w p are respectively the macro variable representations of the batch pose of the camera to be optimized, the batch poses of all dynamic objects, the world coordinate system coordinates of static feature points, and the world coordinate system coordinates of dynamic feature points. Σ r and Σ o are respectively the covariance matrices of the measurement noises of static feature points and dynamic feature points.

6. The dense reconstruction method for a static environment and dynamic objects according to claim 5, characterized in that, Back-project the static pixel points on the depth map into the camera coordinate system and use the optimized camera pose to transform it into the world coordinate system, so as to establish a dense point cloud map for the static environment. Specifically: For the pixels outside the edge mask on the static depth map, back-project them into the camera coordinate system and use the optimized camera pose to transform them into the world coordinate system, that is: Among them, and respectively represent the homogeneous coordinates of the i-th feature point in the k-th frame of the static image in the pixel coordinate system and the coordinates in the world coordinate system. Z represents the depth value of the feature point in the camera coordinate system, and K represents the internal parameter matrix under the camera pinhole model. After coordinate transformation, the color of the point cloud can be obtained from the corresponding pixels on the RGB image. By performing the above transformation on the available pixels on the static depth maps of all key frames and fusing the obtained point clouds, a dense static point cloud map is obtained.

7. Dense reconstruction system for static environment and dynamic objects, characterized in that, Applied to the dense reconstruction method of the static environment and dynamic object according to any one of claims 1-6, including an image acquisition module, a first pose calculation module, a second pose calculation module, an optimization module, a depth map processing module, a dense point cloud map construction module, and a dense reconstruction module; The image acquisition module is used to perform instance segmentation on the input image to obtain a static image and an instance image; The first pose calculation module is used to extract and match the first feature points on the static image and calculate the initial pose of the camera based on the first feature points; The second pose calculation module is used to extract and track the second feature points on the instance image and calculate the initial relative pose of the dynamic object based on the second feature points; The optimization module is used to perform tightly coupled non-linear optimization on the constraint relationships of the initial pose of the camera, the initial relative pose of the dynamic object, the coordinates of the first feature points, and the coordinates of the second feature points. The tightly coupled non-linear optimization includes the pose constraint relationship of the camera and the relative pose constraint relationship of the dynamic object; The depth map processing module is used to obtain a depth map by depth estimation of the input image and preprocess the depth map by using morphological methods and edge detection methods; The dense point cloud map construction module is used to back-project the static pixel points on the depth map into the camera coordinate system and use the optimized camera pose to transform it into the world coordinate system, so as to establish a dense point cloud map for the static environment; The dense reconstruction module is used to transform the dynamic object pixel points on the depth map into the world coordinate system, obtain multiple point cloud segments of the same object, screen the point cloud segments, and finally use the optimized relative pose of the dynamic object to transform the retained point cloud segments to the reference segment for fusion, so as to densely reconstruct the dynamic object.

8. A computer-readable storage medium storing a program, characterized in that, When the program is executed by a processor, it implements the dense reconstruction method of the static environment and dynamic object according to any one of claims 1-6.

Images