A method for detecting moving objects in a three-dimensional scene

Abstract

The application discloses a three-dimensional scene moving object detection method, which comprises the following steps: firstly, preprocessing point cloud models of the same scene created at two time points, then registering the two point clouds after preprocessing; carrying out object detection on the two registered point clouds respectively to obtain two object sets; then, carrying out difference on the two registered point clouds to obtain their difference regions, extracting two candidate moving object sets from the two object sets according to the difference regions; finally, carrying out violent registration object matching on the candidate moving object sets to obtain final moving information. The method is more lightweight and simple, and is convenient for use in public security, security and protection, robots and the like.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a method for detecting and estimating moving objects. Technical Background

[0002] With the development of computer vision technology, 3D vision has been widely applied, including unmanned vehicles at Cainiao Post Stations, military and civilian drones, robots with various functions, and the development of VR and AR, all demonstrating the practicality and advancement of 3D vision. Initially, only large LiDAR sensors could collect data; then came small 3D scanners, and then handheld depth cameras, giving us more possibilities to acquire 3D data using sensors. Point clouds are key information collected by sensors. A point cloud is a collection of points. Compared to images, point clouds have an irreplaceable advantage—depth. Point clouds retain the original geometric information in 3D space without discretization, preserving a large amount of scene information, making them ideal input for 3D data. This is why more and more 3D vision projects use point clouds as input. However, some applications of 3D vision are still relatively lacking; there is no mature technology for recognizing changes in scenes in 3D space, and more and more fields are placing higher demands on it, such as in public security, security, and surveillance. We propose a new method for detecting moving objects in 3D scenes. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a method for detecting moving objects in a three-dimensional scene.

[0004] To enable the detection of movement of planar objects, this invention proposes a method for detecting changes in the position and estimating the pose of objects on a plane within the same scene. The input is a point cloud on a plane within the same scene, and the output is an estimate of the changes in objects on the plane within the scene (including translation, rotation, and their combinations, or disappearance and appearance).

[0005] A method for detecting moving objects in a 3D scene includes the following steps:

[0006] Step (1): Preprocess the point cloud models created at two time points for the same scene;

[0007] Step (2): Register the two preprocessed point clouds;

[0008] Step (3): Perform object detection on the two registered point clouds to obtain two object sets;

[0009] Step (4): Difference between the two registered point clouds to obtain their difference regions, and extract two candidate moving object sets from the two object sets according to the difference regions.

[0010] Step (5): Perform brute-force registration and object matching on the candidate moving object set to obtain the final moving information.

[0011] The beneficial effects of this invention are as follows:

[0012] We propose methods such as planar projection to complete objects and point cloud closure to determine objects, enabling computational processing of two point clouds in the same scene. This allows for the detection of planar objects and estimation of positional changes of changing objects. Compared with previous methods, our method is more lightweight and concise, making it easier to apply in public security, security, robotics, and other fields. Attached Figure Description

[0013] Figure 1 This is a flowchart of object movement detection according to an embodiment of the present invention;

[0014] Figure 2 This is a schematic diagram of the spherical division plane unfolding according to an embodiment of the present invention. Detailed Implementation

[0015] The technical solution of the present invention will be further defined below with reference to the accompanying drawings and embodiments.

[0016] like Figure 1 As shown, a method for detecting moving objects in a 3D scene includes the following steps:

[0017] Step (1) Preprocess the point cloud models created for the same scene at two different time points;

[0018] 1.1 Obtaining 3D point cloud models P of the same scene at two different times n and P o ;

[0019] 1.2 First, consider the two point clouds P n and P o Voxel filtering was performed separately, with each voxel being 1 cm in size, to obtain the filtered point cloud P. n * and P o *;

[0020] 1.3 Filtered point cloud P n * and P o * The normal vector is calculated using Principal Component Analysis (PCA). The calculation requires a neighborhood of 10 points, resulting in the filtered point cloud P. n * and P o * normal vector N n and N o ;

[0021] 1.4 Filtered point cloud P n * and Po * The curvature is calculated using the principal curvature estimation method to obtain the filtered point cloud P. n * and P o * curvature C n and C o ;

[0022] Step (2) Register the two preprocessed point clouds;

[0023] 2.1 Point cloud P n Extract Harris keypoints to obtain keypoint set K. n The Gaussian curvature C of these key points is calculated using the neighboring point normal vector estimation method. n For the key point set K n Remove Gaussian curvature C n Keypoints with a value less than 0.08 are considered keypoints; the remaining set of keypoints is called K. n * Then, for the set of key points K s *Extract FPFH descriptor D n *. For point cloud P o Perform the same operation to obtain the filtered keypoint set K. o * and descriptor D o *;

[0024] 2.2 Using the key point set K n *、K o * and descriptor D n *、D o * Coarse registration was performed using the TEASER++ algorithm to obtain the point cloud P. o To P n Transformation matrix T c ;

[0025] 2.3 Using the filtered point cloud P n * and P o * normal vector N n and N o The point cloud P is processed using the Iterative Closest Point (ICP) algorithm, which uses a point-to-plane approach. n To P o Perform fine registration to obtain the transformation matrix T j The initial value of the ICP algorithm is T. c ;

[0026] 2.4 Point cloud P o Transform to point cloud P using the following formula. n Using the coordinate system, we obtain the point cloud P. Oz :

[0027] p Oz =T j *T c *p o

[0028] Among them, P Oz Represents point cloud P Oz For each 3D point (using homogeneous coordinates) in the array, p o Represents point cloud P o Each three-dimensional point in the array (using homogeneous coordinates);

[0029] Step (3) Perform object detection on the two registered point clouds to obtain two object sets;

[0030] 3.1 Point cloud P Oz The plane is extracted using the Random Sample Consensus (RANSAC) method, resulting in a set of planar point clouds and the corresponding plane equations.

[0031] {(P O_pi M O_pi ), i = 1, 2…N O_pi}

[0032] The set of planar point clouds is abbreviated as P. o_p The set of equations for a planar point cloud is denoted as M. o_p Where, N O_pi Represents point cloud P Oz The number of planes in P O_pi Let M represent the 3D point cloud of the i-th plane. O_pi Let P represent the equation of the i-th plane. The RANSAC method iterates 500 times with a neighborhood threshold of 2 cm, requiring the plane to contain 1000-10000 points. After extracting the planes, the point cloud P is... Oz The remaining point cloud is called P. O '.

[0033] P for point cloud n Performing the same operation yields the set of planar point clouds and the corresponding plane equations for the planar point clouds:

[0034] {(P n_pj M n_pj ), j = 1, 2, ..., N n_pj}

[0035] The set of planar point clouds is abbreviated as P. n_p The set of equations for a planar point cloud is denoted as M. n_p Where, N n_pj Represents point cloud P n The number of planes in P n_pjLet M represent the 3D point cloud of the j-th plane. n_pj Let P represent the equation of the j-th plane. The RANSAC method iterates 500 times with a neighborhood threshold of 2 cm, requiring the plane to contain between 1000 and 10000 points. After extracting the planes, the point cloud P... n The remaining point cloud is called P. n '.

[0036] 3.2 P O Perform Euclidean clustering with a clustering threshold of 2 cm to obtain a point cloud set of candidate objects {P}. o_obm m = 1, 2…N O_ob}, abbreviated as P O_ob The m-th category is P. O_obm N O_ob Represents the number of categories, and calculates P. O_ob The centroid coordinates Cen of each point cloud element O_m .

[0037] For P n Perform Euclidean clustering with a clustering threshold of 2 cm to obtain a point cloud set of candidate objects {P}. n_obk k = 1, 2, ..., N n_ob Abbreviated as P n_ob The k-th category is P n_obk N n_ob Represents the number of categories, and calculates P. n_ob The centroid coordinates Cen of each point cloud element n_k .

[0038] 3.3 Point cloud set P n_ob and P O_ob To determine whether something is an object, perform the following operations:

[0039] The following uses point cloud set P O_ob For example, for P n_ob The judgment method is the same.

[0040] 3.3.1 Calculate and determine the point cloud set P O_ob With point cloud set P O_p Whether it borders another country. The calculation method is as follows:

[0041] For point cloud P O_obm and P O_pi Each three-dimensional point p O_obm and p O_pi ,

[0042]

[0043] The `Adjacent` function determines whether two input point clouds are adjacent; a value of 1 indicates they are adjacent, and a value of 0 indicates they are not. Find P O_pi After the plane of adjacency is connected, the projection point cloud P is obtained by projecting onto the plane of adjacency. t Then P O_pi Complete the equation to get P O_pI P O_pI =P O_pi +P t Complete the object.

[0044] 3.3.2 Based on point cloud P O_pI The closure property determines whether it is an object.

[0045] For P O_pI Normalize the point cloud to a centroid sphere. The normalization calculation method is to first calculate the point cloud P. O_pI The centroid (the formula for calculating the centroid can be seen in step 4.2) is recorded as p. Io For point cloud P O_pI Each point p O_pI Normalized calculation:

[0046] Obtain the spherical normalized point cloud P O_pI '. The point cloud P O_pI 'Divided according to the spherical coordinate system, where any point p in the point cloud...' O_pI The coordinates of ' are (r, θ, φ), and given that r = 1, the partitioning method is shown in [reference needed]. Figure 2 (Reference: Malkin ZA new method to subdivide a spherical surface into equal-area cells[J]. arXiv preprint arXiv: 1612.03467, 2016). The specific subdivisions for θ = 0-10.2148°, 10.2148°-20.7738°, 20.7738°-30.1631°…, 70.2010°-80.1375°, 80.1375°-90° are shown in the table below. The blocks are divided into 10, 10, 12...40, 120 blocks as shown in the table, for a total of 406 blocks;

[0047]

[0048] Convert the 3D coordinates of each point in the point cloud to spherical coordinates, and then count the number of points in each block, denoted as N. O_pI If the number of points in all 406 regions is greater than 500, then the point cloud is an object. The Close() function can determine whether a point cloud is closed. It returns 1 if the point cloud is closed and 0 if it is not closed. If the point cloud is closed, it is an object.

[0049] Save the obtained object collection {P O_obz z = 1, 2, ..., O}, P O_obz The point cloud represents an object, and there are O objects; similarly, for the point cloud set P... n_ob Perform steps 3.3.1 and 3.3.2 to obtain the object set {P}. n_obq , q=1,2…,a}, P n_obq Let a represent the point cloud of an object, which has a points.

[0050] Step (4) Difference is performed on the two registered point clouds to obtain their difference regions. Based on the difference regions, two candidate moving object sets are extracted from the two object sets respectively.

[0051] 4.1 Calculate the point cloud difference for P n_obq and P O_obz The difference formula for point clouds is as follows: P nO =Diff(P n_obq P O_obz );

[0052] Where P nO It is P n_obq For P O_obz The point cloud after difference, where the lowercase letter p represents a 3D point. P On =Diff(P O_obz P n_obq );

[0053] Where P On It is P O_obz For P n_obq The point cloud after difference, where the lowercase letter 'p' represents a 3D point.

[0054] P for point cloud nO Perform Euclidean clustering with a clustering threshold of 2cm to obtain a set of candidate object point clouds P. nO ', P nOu Let u be the point cloud;

[0055] P for point cloud On Perform Euclidean clustering with a clustering threshold of 2cm to obtain a set of candidate object point clouds P. On ', P Ong Let g be the g-th point cloud.

[0056] 4.2 Find the set P of moving objects O_mv and Pn _ mv ;{PO_mv}={P l}, P l ∈{P O_obz},right

[0057] Among them, P O_mv It is P O_obz For moving objects, Cen represents the centroid coordinates of the calculated point cloud.

[0058] {P n_mv}={P L}, P L ∈{P n_obq},right

[0059] Among them, P n_mv It is P n_obq Moving objects.

[0060] Step (5) Perform brute-force registration and object matching on the candidate moving object set to obtain the final movement information;

[0061] 5.1 P n_mv Point clouds and P in O_mv The point cloud is brute-force matched, and the minimum root mean square error (RMSE) rmse between each pair is calculated. If rmse < 0.01, the object is successfully matched. For each successfully matched pair of object point clouds, the Iterative Closest Point (ICP) algorithm is used to output and save the transformation matrix M.

[0062] 5.2 From P n_obq Remove the matching object point clouds from the point cloud, and the remaining objects are the newly added objects, starting from P. O_obg Similarly, remove the matching object points from the point cloud, and the remaining objects are the newly added objects.

[0063] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0064] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A method for detecting moving objects in a three-dimensional scene, characterized in that, The steps include the following: Step (1): Preprocess the point cloud models created at two time points for the same scene; Step (2): Register the two preprocessed point clouds; Step (3): Perform object detection on the two registered point clouds to obtain two object sets; Step (4): Difference between the two registered point clouds to obtain their difference regions, and extract two candidate moving object sets from the two object sets according to the difference regions; Step (5): Perform brute-force registration and object matching on the candidate moving object set to obtain the final movement information; The specific method for step (1) is as follows; 1.1 Obtaining 3D point cloud models of the same scene at two different times. and ; 1.2 First, analyze the two point clouds. and Voxel filtering was performed separately, with each voxel being 1 cm in size, to obtain the filtered point cloud. and ; 1.3 Filtered point cloud and The normal vector is calculated using principal component analysis, requiring a neighborhood of 10 points to obtain the filtered point cloud. and normal vector and ; 1.4 Filtered point cloud and The curvature is calculated using the principal curvature estimation method to obtain the filtered point cloud. and curvature and ; The specific method for step (2) is as follows; 2.1 Point Clouds Extract Harris keypoints to obtain a keypoint set. The Gaussian curvature of these key points was calculated using the neighboring point normal vector estimation method. For the set of key points Remove Gaussian curvature Keypoints with a value less than 0.08 are considered keypoints; the remaining set of keypoints is called... Then for the set Key point extraction of FPFH descriptors Point clouds Perform the same operation to obtain the filtered set of key points. and descriptors ; 2.2 Using a set of key points , and descriptors , The point cloud was obtained by coarse registration using the TEASER++ algorithm. arrive Transformation matrix ; 2.3 Utilizing the filtered point cloud and normal vector and The point cloud is processed by the iterative nearest point algorithm from point to surface. arrive Perform fine registration to obtain the transformation matrix. The initial value of the ICP algorithm is ; 2.4 Point Cloud Transform to point cloud using the following formula. Using the coordinate system, we obtain the point cloud. : in, Point cloud Each three-dimensional point in the image, Point cloud Each three-dimensional point in the array.

2. The method for detecting moving objects in a three-dimensional scene according to claim 1, characterized in that, The specific method for step (3) is as follows; 3.1 Point Clouds The plane is extracted using the Random Sample Consensus (RANSAC) method, resulting in a set of planar point clouds and the corresponding plane equations. The planar point cloud set is abbreviated as The set of equations for a planar point cloud is denoted as in, Point cloud The number of planes in the middle, Represents the 3D point cloud of the i-th plane. Let represent the equation of the i-th plane; the number of iterations for extracting the plane using RANSAC on the point cloud is 500, the neighborhood threshold for the distance from a point to the plane is 2 cm, and the number of 3D points contained in the extracted plane is required to be 1000-10000; after extracting the plane, the point cloud... The remaining point cloud is called ; Point cloud Performing the same operation yields the set of planar point clouds and the corresponding plane equations for the planar point clouds: The planar point cloud set is abbreviated as The set of equations for a planar point cloud is denoted as in, Point cloud The number of planes in the middle, Represents the 3D point cloud of the j-th plane. Let represent the equation of the j-th plane; the RANSAC method is used to extract the planes from the point cloud for 500 iterations, with a neighborhood threshold of 2 cm for the distance from a point to the plane, and the extracted planes are required to contain 1000-10000 3D points; after extracting the planes, the point cloud... The remaining point cloud is called ; 3.2 (Regarding) Euclidean clustering with a clustering threshold of 2 cm was performed to obtain a set of candidate object point clouds. Abbreviated as The m-th category is , Represents the number of categories, and calculates... The centroid coordinates of each point cloud element ; right Euclidean clustering with a clustering threshold of 2 cm was performed to obtain a set of candidate object point clouds. Abbreviation The kth category is , Represents the number of categories, and calculates... The centroid coordinates of each point cloud element ; 3.3 Point Cloud Aggregation and To determine if something is an object, perform the following steps: The following uses point cloud collections For example, for Judgment methods and The judgment method is the same; 3.3.1 Calculate and determine the point cloud set With point cloud collection Whether they are adjacent; the calculation method is as follows: The `Adjacent` function is used to determine whether two input point clouds are adjacent; a value of 1 indicates they are adjacent, and a value of 0 indicates they are not adjacent. After the contiguous plane is formed, a point cloud is obtained by projecting the point cloud onto the contiguous plane. Then on Complete to get , Complete the object; 3.3.2 Based on point clouds The closure property determines whether it is an object; Point cloud Perform centroid spherical normalization; the normalization calculation method is to first calculate the point cloud. The center of mass is recorded as For point clouds every point Normalized calculation: , obtain spherical normalized point cloud ; to point clouds Based on the spherical coordinate system, any point in the point cloud... The coordinates are (r, θ, φ), and r = 1. For θ = 0 - 10.2148°, 10.2148° - 20.7738°, 20.7738° - 30.1631°, 30.1631° - 40.5602°, 40.5602° - 50.2170°, 50.2170° - 60.1113°, 60.1113° - 70.2010°, 70.2010° - 80.1375°, 80.1375° - 90°, φ is divided into 10, 10, 12, 12, 15, 18, 24, 40, and 120 blocks respectively, for a total of 406 blocks. Convert the 3D coordinates of each point in the point cloud to spherical coordinates, and then count the number of points in each block, denoted as . If the number of points in all 406 regions is greater than 500, then the point cloud is an object. The Close() function is used to determine whether a point cloud is closed. It returns 1 to indicate that the point cloud is closed and 0 to indicate that it is not closed. It determines whether a closed point cloud is an object. Save the collection of objects , The point cloud represents an object, and there are O objects; similarly, for the point cloud set... Perform steps 3.3.1 and 3.3.2 to obtain a set of objects. , Let a represent the point cloud of an object, which has a points.

3. The method for detecting moving objects in a three-dimensional scene according to claim 2, characterized in that, The specific method for step (4) is as follows; 4.1 Calculate the point cloud difference, for the point cloud The difference formula is as follows: in yes The point cloud after difference, where the lowercase letter p represents a 3D point; in yes The point cloud after difference, where the lowercase letter p represents a 3D point; Point cloud Euclidean clustering with a clustering threshold of 2cm was performed to obtain a set of candidate object point clouds. , ; Point cloud Euclidean clustering with a clustering threshold of 2cm was performed to obtain a set of candidate object point clouds. , ; 4.2 Find the set of moving objects and ; in, yes For moving objects, Cen represents the centroid coordinates of the calculated point cloud; in, yes Moving objects.

4. The method for detecting moving objects in a three-dimensional scene according to claim 3, characterized in that, The specific method for step (5) is as follows; 5.1 will Point clouds and The point cloud is brute-force matched, and the minimum root mean square error rmse between each pair is calculated. If rmse < 0.01, the object is successfully matched. For each successfully matched pair of object point clouds, the iterative nearest point algorithm is used to output and save the transformation matrix M. 5.2 From Remove the matching object point clouds from the point cloud; the remaining objects are the newly added objects. Similarly, remove the matching object point clouds from the point cloud, and the remaining objects are the newly added objects.

Images