Outdoor Static Environment Modeling Method and Device Based on Point Cloud Bird's-eye View

Abstract

This invention discloses a method and apparatus for modeling outdoor static environments based on point cloud bird's-eye view. The method includes: acquiring outdoor static environment point cloud data; performing point cloud coordinate system transformation based on the acquired outdoor static environment point cloud data to obtain a point cloud under the vertical visual perspective of the monitoring area; generating a point cloud bird's-eye view based on the point cloud under the vertical visual perspective of the monitoring area; labeling the regional attributes in the point cloud bird's-eye view to obtain semantic information in the monitoring environment; and generating an outdoor static area model based on the semantic information. This invention is simple and efficient, realizing the extraction of dense environmental information from discrete point cloud information, efficiently extracting dynamic objects, and transforming disordered point cloud data into ordered point cloud data, making it more effective in empowering upper-layer applications. It can be widely applied in the field of point cloud data processing technology.

Description

Technical Field

[0001] This invention relates to the field of point cloud data processing technology, and in particular to a method and apparatus for modeling outdoor static environments based on point cloud bird's-eye views. Background Technology

[0002] Currently, most dynamic target recognition in outdoor environments relies on video equipment. This involves acquiring video image data and then using deep learning-based algorithms to identify dynamic targets. However, video image-based monitoring methods have the following drawbacks:

[0003] 1. Deep learning models suffer from poor generalization. Deep learning models can only recognize pre-trained target categories; they are ineffective for unfamiliar categories. Furthermore, their recognition performance is often poor for specific individuals within certain categories. Therefore, deep learning-based image object recognition algorithms all suffer from poor generalization to varying degrees. This significantly reduces the reliability of dynamic object recognition.

[0004] 2. Poor environmental adaptability, making it difficult to achieve all-weather identification of the monitored area. Image-based methods are greatly affected by weather and lighting conditions. Rainy, foggy weather or poor lighting conditions can significantly impact the performance of deep learning algorithms. Therefore, image-based methods cannot achieve all-weather monitoring, which greatly affects the accuracy of downstream operations and severely limits the applicability of this method.

[0005] 3. Lack of accurate 3D information can easily lead to misjudgments. Video image-based methods struggle to make accurate judgments on a spatial scale, especially at poor angles. This can result in misjudgments of the behavior of dynamic objects, particularly in complex scenarios, which can severely impact downstream business operations.

[0006] The above three points highlight inherent problems that are difficult to solve in image-based dynamic target extraction. Therefore, the industry urgently needs alternatives to vision-based dynamic target extraction solutions to address these pain points.

[0007] Unlike intelligent recognition methods based on video images, laser point clouds are more adaptable to the environment. Point cloud data based on active devices such as lasers are less affected by ambient lighting, thus achieving excellent detection results even at night. This gives the dynamic target extraction scheme based on laser point clouds the advantage of being able to work around the clock. Furthermore, because laser point clouds have accurate three-dimensional information, they can accurately judge the behavior of dynamic targets, such as crossing boundaries or illegal parking.

[0008] However, data processing based on continuous point clouds presents the following two problems:

[0009] 1. Unlike images, point clouds are discrete, and information can be lost in certain areas.

[0010] 2. The amount of continuous point cloud frame data is quite large. In practical applications, how to process and detect it quickly is a significant challenge. Summary of the Invention

[0011] In view of this, embodiments of the present invention provide a simple and efficient method and apparatus for modeling outdoor static environments based on point cloud bird's-eye view.

[0012] One aspect of this invention provides a method for modeling outdoor static environments based on point cloud bird's-eye views, including:

[0013] Acquire outdoor static environment point cloud data;

[0014] The point cloud coordinate system is transformed based on the obtained outdoor static environment point cloud data to obtain the point cloud under vertical vision of the monitoring area.

[0015] A point cloud bird's-eye view is generated based on the point cloud under the vertical vision of the monitored area.

[0016] The regional attributes of the point cloud bird's-eye view are labeled to obtain semantic information in the monitoring environment;

[0017] An outdoor static area model is generated based on the semantic information.

[0018] Optionally, acquiring outdoor static environment point cloud data includes:

[0019] Collect corresponding point cloud data when there are no dynamic objects in the monitored area;

[0020] After collecting data for the monitored area, the point cloud data is edited.

[0021] By statistically analyzing multiple consecutive frames of point clouds, dynamic point clouds can be eliminated.

[0022] Optionally, the step of performing point cloud coordinate system transformation based on the acquired outdoor static environment point cloud data to obtain the point cloud under vertical vision of the monitoring area includes:

[0023] Using data obtained from the inertial testing unit, the angle of tilt of the laser device in the vertical direction is calculated to obtain the rotation matrix required to convert the point cloud into a vertical view;

[0024] or,

[0025] Using a plane in the detection area as a reference plane, calculate the normal vector of the reference plane, calculate the angle between the normal vector of the reference plane and the coordinate axis perpendicular to the original point cloud, and obtain the rotation matrix required to convert the point cloud into a vertical view;

[0026] After obtaining the rotation matrix, the point cloud is transformed from the original coordinate system into a world coordinate system with the z-axis perpendicular to the detection environment.

[0027] Optionally, generating a point cloud bird's-eye view based on the point cloud under the vertical vision of the monitored area includes:

[0028] Configure an origin point in the monitoring area, obtain the corresponding value of the origin point in the point cloud from the bird's-eye view, subtract the point cloud of the origin point from all the points in the point cloud, and convert the coordinate values ​​of all the point clouds in the monitoring area into positive values.

[0029] Set the scale of each pixel in the bird's-eye view corresponding to the monitored area;

[0030] Based on the established correspondence between pixels and real-world scale, and considering the different directions of the point cloud axis and the image axis, the pixel value corresponding to each point cloud in the monitoring environment when projected onto the bird's-eye view is calculated.

[0031] After obtaining the projection relationship between the point cloud and the bird's-eye view, the feature value of each pixel in the bird's-eye view is calculated to generate the point cloud bird's-eye view.

[0032] Optionally, generating an outdoor static area model based on the semantic information includes:

[0033] For all missing points in the point cloud data, the three nearest valid points to the missing point are taken as the neighboring points of the missing point.

[0034] Calculate the distance between neighboring points and missing points. If the distance between all neighboring points and missing points is less than a given distance threshold, and all neighboring points and missing points have the same semantic attributes, then all neighboring points are valid neighboring points; otherwise, all neighboring points are invalid neighboring points.

[0035] Based on the obtained effective nearest neighbor points, calculate the average of the maximum height values ​​of all effective nearest neighbor points as the maximum height value of the missing point, and calculate the average of the minimum height values ​​of all effective nearest neighbor points as the minimum height value of the missing point.

[0036] The corresponding height information is obtained by calculating the effective neighbor points, and the missing points are marked as effective points until all missing points are marked as effective points.

[0037] Optionally, the method further includes: extracting dynamic objects based on the outdoor static area model.

[0038] Another aspect of this invention provides an outdoor static environment modeling device based on point cloud bird's-eye view, comprising:

[0039] The first module is used to acquire outdoor static environment point cloud data;

[0040] The second module is used to perform point cloud coordinate system transformation based on the acquired outdoor static environment point cloud data to obtain the point cloud under vertical vision of the monitoring area.

[0041] The third module is used to generate a point cloud bird's-eye view based on the point cloud under the vertical vision of the monitored area.

[0042] The fourth module is used to annotate the regional attributes of the point cloud bird's-eye view and obtain semantic information in the monitoring environment;

[0043] The fifth module is used to generate an outdoor static area model based on the semantic information.

[0044] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0045] The memory is used to store programs;

[0046] The processor executes the program to implement the method described above.

[0047] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0048] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0049] Embodiments of this invention acquire outdoor static environment point cloud data; perform point cloud coordinate system transformation based on the acquired outdoor static environment point cloud data to obtain a point cloud under the vertical visual perspective of the monitoring area; generate a point cloud bird's-eye view based on the point cloud under the vertical visual perspective of the monitoring area; annotate the regional attributes in the point cloud bird's-eye view to obtain semantic information in the monitoring environment; and generate an outdoor static area model based on the semantic information. This invention is simple and efficient, realizing the acquisition of dense environmental information from discrete point cloud information, efficiently extracting dynamic objects, and transforming disordered point cloud data into ordered point cloud data, making it more effectively empowering upper-layer applications. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 The overall process flowchart provided for embodiments of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] To address the problems existing in the prior art, one aspect of this invention provides a method for modeling outdoor static environments based on point cloud bird's-eye views, such as... Figure 1 As shown, the method of the present invention includes the following steps:

[0054] Acquire outdoor static environment point cloud data;

[0055] The point cloud coordinate system is transformed based on the obtained outdoor static environment point cloud data to obtain the point cloud under vertical vision of the monitoring area.

[0056] A point cloud bird's-eye view is generated based on the point cloud under the vertical vision of the monitored area.

[0057] The regional attributes of the point cloud bird's-eye view are labeled to obtain semantic information in the monitoring environment;

[0058] An outdoor static area model is generated based on the semantic information.

[0059] Optionally, acquiring outdoor static environment point cloud data includes:

[0060] Collect corresponding point cloud data when there are no dynamic objects in the monitored area;

[0061] After collecting data for the monitored area, the point cloud data is edited.

[0062] By statistically analyzing multiple consecutive frames of point clouds, dynamic point clouds can be eliminated.

[0063] Optionally, the step of performing point cloud coordinate system transformation based on the acquired outdoor static environment point cloud data to obtain the point cloud under vertical vision of the monitoring area includes:

[0064] Using data obtained from the inertial testing unit, the angle of tilt of the laser device in the vertical direction is calculated to obtain the rotation matrix required to convert the point cloud into a vertical view;

[0065] or,

[0066] Using a plane in the detection area as a reference plane, calculate the normal vector of the reference plane, calculate the angle between the normal vector of the reference plane and the coordinate axis perpendicular to the original point cloud, and obtain the rotation matrix required to convert the point cloud into a vertical view;

[0067] After obtaining the rotation matrix, the point cloud is transformed from the original coordinate system into a world coordinate system with the z-axis perpendicular to the detection environment.

[0068] Optionally, generating a point cloud bird's-eye view based on the point cloud under the vertical vision of the monitored area includes:

[0069] Configure an origin point in the monitoring area, obtain the corresponding value of the origin point in the point cloud from the bird's-eye view, subtract the point cloud of the origin point from all the points in the point cloud, and convert the coordinate values ​​of all the point clouds in the monitoring area into positive values.

[0070] Set the scale of each pixel in the bird's-eye view corresponding to the monitored area;

[0071] Based on the established correspondence between pixels and real-world scale, and considering the different directions of the point cloud axis and the image axis, the pixel value corresponding to each point cloud in the monitoring environment when projected onto the bird's-eye view is calculated.

[0072] After obtaining the projection relationship between the point cloud and the bird's-eye view, the feature value of each pixel in the bird's-eye view is calculated to generate the point cloud bird's-eye view.

[0073] Optionally, generating an outdoor static area model based on the semantic information includes:

[0074] For all missing points in the point cloud data, the three nearest valid points to the missing point are taken as the neighboring points of the missing point.

[0075] Calculate the distance between neighboring points and missing points. If the distance between all neighboring points and missing points is less than a given distance threshold, and all neighboring points and missing points have the same semantic attributes, then all neighboring points are valid neighboring points; otherwise, all neighboring points are invalid neighboring points.

[0076] Based on the obtained effective nearest neighbor points, calculate the average of the maximum height values ​​of all effective nearest neighbor points as the maximum height value of the missing point, and calculate the average of the minimum height values ​​of all effective nearest neighbor points as the minimum height value of the missing point.

[0077] The corresponding height information is obtained by calculating the effective neighbor points, and the missing points are marked as effective points until all missing points are marked as effective points.

[0078] Optionally, the method further includes: extracting dynamic objects based on the outdoor static area model.

[0079] Another aspect of this invention provides an outdoor static environment modeling device based on point cloud bird's-eye view, comprising:

[0080] The first module is used to acquire outdoor static environment point cloud data;

[0081] The second module is used to perform point cloud coordinate system transformation based on the acquired outdoor static environment point cloud data to obtain the point cloud under vertical vision of the monitoring area.

[0082] The third module is used to generate a point cloud bird's-eye view based on the point cloud under the vertical vision of the monitored area.

[0083] The fourth module is used to annotate the regional attributes of the point cloud bird's-eye view and obtain semantic information in the monitoring environment;

[0084] The fifth module is used to generate an outdoor static area model based on the semantic information.

[0085] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0086] The memory is used to store programs;

[0087] The processor executes the program to implement the method described above.

[0088] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0089] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0090] The specific implementation process of the present invention will now be described in detail with reference to the accompanying drawings:

[0091] 1. Acquire outdoor static environment point cloud data;

[0092] 2. Point cloud coordinate system transformation;

[0093] 3. Point cloud bird's-eye view generation;

[0094] 4. Manual semantic labeling of the monitored area;

[0095] 5. Based on manually labeled semantic information, an outdoor static area model is generated from the bird's-eye view.

[0096] 6. Storage and use of the monitoring area model, and extraction of dynamic objects from the bird's-eye view.

[0097] Step 1: Acquire outdoor static environment point cloud data:

[0098] Regarding the acquisition of point cloud data, firstly, a point cloud is a collection of points in three-dimensional space, an important means of three-dimensional perception, possessing rich geometric and structural information, and has wide applications in fields such as autonomous driving, terrain mapping, and industrial measurement. A point cloud consists of N D-dimensional points. When D=3, it can be represented as a three-dimensional coordinate point (x, y, z). Each point is determined by a certain x, y, or z value. In addition to x, y, and z coordinates, a point cloud can also contain color information, light intensity, category labels, normal vectors, grayscale values, etc. The point cloud described in this invention refers by default to a point cloud containing only x, y, and z coordinate information. Depending on the different application requirements, the source of point clouds can be diverse. Common source sources of point clouds include: LiDAR, cameras with depth sensors, structured light scanning, and 3D reconstruction based on 2D images, etc. These devices or methods can all generate point cloud data.

[0099] To obtain point cloud data of the environment, it is necessary to remove the point clouds of dynamic objects in the monitored area. To obtain a clean point cloud of the monitored area environment, there are generally three methods:

[0100] 1. Collect the corresponding point cloud data only when there are no moving objects in the monitored area;

[0101] 2. After collecting data for the monitored area, use tools such as CloudCompare to manually edit the point cloud data and remove the point clouds of dynamic objects;

[0102] 3. Use algorithms to remove dynamic point clouds, such as by statistically analyzing multiple consecutive frames of point clouds to distinguish between the corresponding dynamic and non-dynamic point clouds.

[0103] Of the three methods above, method 1 is the most direct and efficient. However, due to the complexity of actual application scenarios, it is difficult to guarantee that there will be times when no dynamic objects pass by in every application scenario. Therefore, in practical applications, method 2 and method 3 are usually combined to obtain the static point cloud of the monitoring area.

[0104] Step 2: Point cloud coordinate system transformation:

[0105] As in step 1, the "shooting" angle of the obtained raw point cloud data is in most cases not perpendicular to the monitoring area. Therefore, this embodiment of the invention first needs to convert the angle of the raw environmental point cloud into the angle of the bird's-eye view. This requires converting the coordinate system of the raw point cloud into the world coordinate system under the vertical vision of the monitoring area. At this time, this embodiment of the invention needs to calculate the corresponding rotation matrix. The following two methods are commonly used:

[0106] Method 1: Using data provided by the IMU (Inertial Measurement Unit), calculate the angle of tilt of the laser device in the vertical direction (relative to the ground of the detection area), thereby obtaining the rotation matrix required to convert the point cloud into a vertical view.

[0107] The IMU can output three-axis acceleration and three-axis angular velocity data based on its own coordinate system, and obtain the pose trajectory of the carrier motion through the inertial navigation calculation principle.

[0108] Method 2: Using a plane (usually the ground) in the detection area as a reference plane, calculate the normal vector of the reference plane. The direction pointed to by the reference plane normal vector is the vertical direction of the point cloud bird's-eye view. Calculate the angle between the reference plane normal vector and the coordinate axis of the original point cloud's vertical direction, thus obtaining the rotation matrix required to convert the point cloud into a vertical view.

[0109] In calculating the reference plane normal vector, this invention commonly uses the Random Sample Consensus (RANSAC) algorithm. RANSAC iteratively estimates the parameters of a mathematical model from a set of observed data containing outliers. RANSAC is a non-deterministic algorithm; in a sense, it produces a reasonable result with a certain probability, and more iterations increase this probability. Therefore, in practical engineering, to obtain a more accurate bird's-eye view, multiple reference planes can be selected, the normal vector of each reference plane can be calculated, and then the average can be used to obtain a more accurate vertical vector for the bird's-eye view, thus obtaining the rotation matrix required to convert the point cloud into a vertical view.

[0110] After obtaining the corresponding rotation matrix, the point cloud can be transformed from the original coordinate system into the z-axis (the specific point cloud axis obtained by different devices will be different; here, the z-axis refers to the axis closest to the vertical direction of the monitoring area) perpendicular to the world coordinate system of the detection environment. Then, after projecting along the z-axis direction of the point cloud, the bird's-eye view desired in the embodiment of the present invention will be obtained.

[0111] Step 3: Generating point cloud bird's-eye view:

[0112] A point cloud bird's-eye view refers to the projection of a point cloud onto a plane perpendicular to its height. After converting the point cloud into a point cloud under the vertical vision of the monitoring area in step 2, in order to obtain the bird's-eye view of the monitoring environment required by this embodiment of the invention, this embodiment of the invention needs to project the point cloud in the vertical direction. This is a process of converting a three-dimensional point cloud into a two-dimensional one, and also a process of ordering a disordered point cloud, which is the basis of this invention. First, this embodiment of the invention needs to understand the difference between a bird's-eye view (image) and a three-dimensional point cloud:

[0113] 1. Regarding the bird's-eye view (image): The coordinate values ​​of the image are always positive; the origin of the image coordinate system is at the top left corner of the image; the coordinate values ​​are integers; the x-axis is to the right, and the y-axis is downward.

[0114] 2. Regarding point cloud coordinates: Point cloud coordinate values ​​can be positive or negative; point cloud values ​​are actual numerical values; the x-axis is to the right, the y-axis is forward, and the z-axis is upward (the coordinate axes of point clouds obtained by different devices are different).

[0115] Based on the above features, a bird's-eye view is generated through the following steps:

[0116] (1) Set an origin in the monitoring area, obtain the value of the origin in the point cloud from the bird's-eye view, and subtract the origin point from all the points in the point cloud to convert the x and y values ​​of all the points in the monitoring area into positive values.

[0117] (2) Set the scale of each pixel in the bird's-eye view to correspond to the monitoring area. For example, in real-world applications, one pixel in the bird's-eye view can be set to represent a 0.1m x 0.1m area in the monitoring area.

[0118] (3) Based on the set correspondence between pixels and real scale, and combined with the different directions of point cloud axis and image axis, calculate the corresponding pixel value when each point cloud in the monitoring environment is projected onto the bird's-eye view.

[0119] It should be noted that the size of the bird's-eye view is related to the set pixel ratio and the corresponding ratio of the real environment, as well as the set size of the monitoring area. For example, assuming that one pixel in the bird's-eye view represents a 0.1m*0.1m area in the monitoring area, and the size of the monitoring area is 150m long and 100m wide, then the size of the bird's-eye view obtained from the point cloud of that area is 1500*1000.

[0120] Assuming one pixel in the bird's-eye view represents a 0.1m x 0.1m area within the monitored region, with the image axis x-axis to the right and y-axis downwards, and the point cloud axis x-axis to the right, y-axis forwards, and z-axis upwards, and assuming the monitored region is 150m long and 100m wide, then the pixel x-value corresponding to the point cloud midpoint A(2,3.4,1) projected onto the bird's-eye view is 2 / 0.1 = 20, and the corresponding pixel y-value is (100 / 0.1) - (3.4 / 0.1) = 966. Therefore, the pixel value corresponding to the point cloud midpoint A(2,3.4,1) is (20, 966). When the calculated pixel value is a decimal, it can be rounded to the nearest integer using a uniform scale.

[0121] (4) After obtaining the projection relationship between the point cloud and the bird's-eye view, it is necessary to calculate the feature value of each pixel in the bird's-eye view. This invention is based on the environmental height modeling under the bird's-eye view vision. Therefore, the attribute of each pixel in the bird's-eye view is the height information of the monitoring environment corresponding to that pixel. Among them, the relationship between the bird's-eye view pixel and the point cloud is one-to-many. In the modeling process of this invention, the height information required is the maximum and minimum values ​​on the z-axis (the vertical direction of the environment) of all the point clouds corresponding to that pixel.

[0122] After this step, the embodiment of the present invention has successfully modeled the monitoring area based on the bird's-eye view using height information. At this time, the environmental model only has two pieces of information: the maximum height value and the minimum height value. If a 0.1m*0.1m area in the monitoring area is converted into one pixel, and the size of the monitoring area is 150m by 100m, then the environmental model after modeling is a 1500*1000*2 matrix.

[0123] Step 4: Manual semantic labeling of the monitored area:

[0124] After step 3, the embodiment of the present invention obtains a height information model under the bird's-eye view of the monitoring environment. After obtaining the bird's-eye view, the embodiment of the present invention needs to manually calibrate the bird's-eye view with the help of calibration data in order to obtain richer semantic information in the monitoring environment.

[0125] In step 3, after obtaining the environmental bird's-eye view from the point cloud, this embodiment of the invention uses data annotation software to annotate the regional attributes in the bird's-eye view. The corresponding attributes are defined according to the needs of specific applications.

[0126] In this embodiment of the invention, regions are defined using polygons. Next, this embodiment of the invention only needs to calculate which polygon region each pixel in the bird's-eye view belongs to in order to obtain the semantic information of each pixel in the bird's-eye view. In step 3, this embodiment of the invention obtains the correspondence between point cloud and bird's-eye view pixels. Thus, this embodiment of the invention can associate point cloud information with semantic information. Therefore, this embodiment of the invention can obtain the environmental semantic information of each point in the point cloud through this series of correspondences.

[0127] Step 5: Based on manually calibrated semantic information, generate an outdoor static area model from the bird's-eye view:

[0128] After the preceding steps, this embodiment of the invention has initially obtained a monitoring environment model based on bird's-eye view. This model includes vertical height information and corresponding semantic information. However, in this embodiment, since the point cloud information is discrete and sparse, black areas lack corresponding point cloud information. Therefore, the lack of information in these areas will cause significant problems in practical applications. Thus, this embodiment needs to utilize the original model height information to transform the model from sparse to dense. Since environmental semantics are manually defined through bounding boxes, even pixels without corresponding point cloud data will be bounded and assigned the correct environmental semantics. Therefore, at this stage, the entire model lacks height information. The next question is how this embodiment can utilize the existing height information from the bird's-eye view to calculate the missing height information.

[0129] The specific method of this invention is as follows:

[0130] For bird's-eye view pixels that lack valid height information, this embodiment of the invention refers to them as missing pixels; for bird's-eye view pixels that have valid height information, this embodiment of the invention refers to them as valid pixels.

[0131] 1. For all missing points, obtain the three nearest valid points as the neighbors of the missing point;

[0132] 2. Calculate the distance between neighboring points and missing points. If the distance between all neighboring points and missing points is less than the given distance threshold, and the semantic attributes of all neighboring points and missing points are consistent, then all neighboring points are valid neighboring points. If one or more neighboring points do not meet the above requirements, then all neighboring points are invalid neighboring points.

[0133] 3. If valid neighboring points are obtained after step 2, the average of the maximum height values ​​of all valid neighboring points is calculated as the maximum height value of the missing point, and the average of the minimum height values ​​of all valid neighboring points is calculated as the minimum height value of the missing point; if no three valid neighboring points are obtained after step 2, the information of the missing point will not be backfilled in this algorithm round, and the missing point will remain a missing point.

[0134] 4. After step 3, once the corresponding height information is obtained through the calculation of the effective neighboring points, the missing point is marked as a valid point in the next calculation round.

[0135] 5. Repeat steps 1 to 4 until all missing points are marked as valid points.

[0136] Thus, the monitoring environment modeling of this embodiment of the invention is completed. If a 0.1m*0.1m area in the monitoring area is converted into a pixel, and the size of the monitoring area is 150m by 100m, and each corresponding bird's-eye view pixel in the manually labeled monitoring area has only one corresponding environmental semantic, then the modeled environment is a 1500*1000*3 matrix, which contains the highest and lowest height values ​​of each corresponding location in the environment, as well as the semantic information at the corresponding location.

[0137] Step 6: Storing and using the monitoring area model, and extracting dynamic objects from the bird's-eye view.

[0138] As can be seen from step 5, after modeling the environment based on the bird's-eye view, the embodiment of the present invention obtains an X*Y*N matrix, where the size of X and Y is determined by the defined correspondence between the monitoring area and the bird's-eye view pixel values ​​and the size of the monitoring area, and the value of N is equal to the number of environmental semantics corresponding to each pixel defined by the user plus 2 (the maximum and minimum height values ​​in the vertical direction of the point). For the storage of the X*Y*N matrix of the model, the embodiment of the present invention can store it as a file, such as by using the storage and loading related interfaces of the NumPy module, the calculated environment model can be stored quickly, and it can be loaded very conveniently when needed.

[0139] Taking dynamic object filtering as an example, when the embodiment of the present invention obtains each frame of point cloud data in real time, the embodiment of the present invention first calculates the corresponding pixel points of all points in the point cloud frame on the bird's-eye view of the monitoring area. Through the environmental height information recorded in the model, it is determined whether the point belongs to the environment or is a dynamic point. If the height value of the point is between the maximum height value and the minimum height value of the corresponding point in the model, then the point belongs to the environment of the monitoring area. If it is not within this height range, then the point is a dynamic point in the monitoring area. In this way, the embodiment of the present invention can filter out dynamic points in the monitoring environment very quickly and accurately. By using clustering methods such as DBSAN, the dynamic points are clustered into dynamic targets. Thus, the embodiment of the present invention successfully realizes dynamic target extraction based on point cloud bird's-eye view.

[0140] In addition, the environmental model based on the bird's-eye view assigns a corresponding pixel value to each point cloud point. Unlike the point cloud, the pixel values ​​are ordered. This indicates that, through this invention, the embodiments of this invention can transform point cloud points from disordered to ordered. The ordered point cloud points can better empower subsequent applications. For example, the clustering method based on ordered points has a significant advantage in efficiency compared to the clustering method based on disordered points.

[0141] In summary, this invention innovatively proposes an outdoor static environment modeling method based on laser point cloud bird's-eye view. By establishing an environment model, dense environmental information is obtained from discrete point cloud data, thereby achieving efficient and reliable dynamic target extraction based on laser point clouds.

[0142] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0143] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0144] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0146] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0147] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0148] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0149] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0150] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for outdoor static environment modeling based on point cloud bird's eye view, characterized in that, include: Acquire outdoor static environment point cloud data; The point cloud coordinate system is transformed based on the obtained outdoor static environment point cloud data to obtain the point cloud under vertical vision of the monitoring area. The point cloud under vertical vision of the monitoring area is a world coordinate system with the z-axis perpendicular to the detection environment. A point cloud bird's-eye view is generated based on the point cloud under the vertical vision of the monitored area. The regional attributes of the point cloud bird's-eye view are labeled to obtain semantic information in the monitoring environment; Based on the semantic information, an outdoor static area model is generated; The step of generating a point cloud bird's-eye view based on the point cloud under the vertical vision of the monitored area includes: Configure an origin point in the monitoring area, obtain the corresponding value of the origin point in the point cloud from the bird's-eye view, subtract the point cloud of the origin point from all the points in the point cloud, and convert the coordinate values ​​of all the point clouds in the monitoring area into positive values. Set the scale of each pixel in the bird's-eye view corresponding to the monitored area; Based on the established correspondence between pixels and real-world scale, and considering the different directions of the point cloud axis and the image axis, the pixel value corresponding to each point cloud in the monitoring environment when projected onto the bird's-eye view is calculated. The relationship between the pixels in the bird's-eye view and the point cloud is one-to-many. After obtaining the projection relationship between the point cloud and the bird's-eye view, the feature value of each pixel in the bird's-eye view is calculated to generate the point cloud bird's-eye view. The feature values ​​include the maximum and minimum height values ​​in the corresponding point cloud. Dynamic object extraction is performed based on the outdoor static area model, including: When a frame of point cloud data is obtained, the corresponding pixel points of all points in the corresponding frame are calculated on the bird's-eye view, and the height value of the pixel points is determined. Compare the height value of each pixel with the feature value of the corresponding pixel in the point cloud bird's-eye view; When the height value is outside the maximum and minimum height values ​​of the corresponding pixel, the corresponding pixel is a dynamic point, and the dynamic object is determined based on the dynamic point by the clustering method.

2. The method of claim 1, wherein, The acquisition of outdoor static environment point cloud data includes: Collect corresponding point cloud data when there are no dynamic objects in the monitored area; After collecting data for the monitored area, the point cloud data is edited. By statistically analyzing multiple consecutive frames of point clouds, dynamic point clouds can be eliminated.

3. The method of claim 1, wherein, The step of performing point cloud coordinate system transformation based on the acquired outdoor static environment point cloud data to obtain the point cloud under vertical vision of the monitoring area includes: Using data obtained from the inertial testing unit, the angle of tilt of the laser device in the vertical direction is calculated to obtain the rotation matrix required to convert the point cloud into a vertical view; or, Using a plane in the detection area as a reference plane, calculate the normal vector of the reference plane, calculate the angle between the normal vector of the reference plane and the coordinate axis perpendicular to the original point cloud, and obtain the rotation matrix required to convert the point cloud into a vertical view; After obtaining the rotation matrix, the point cloud is transformed from the original coordinate system into a world coordinate system with the z-axis perpendicular to the detection environment.

4. The method of claim 1, wherein, The step of generating an outdoor static area model based on the semantic information includes: For all missing points in the point cloud data, the three nearest valid points to the missing point are taken as the neighboring points of the missing point. Calculate the distance between neighboring points and missing points. If the distance between all neighboring points and missing points is less than a given distance threshold, and all neighboring points and missing points have the same semantic attributes, then all neighboring points are valid neighboring points; otherwise, all neighboring points are invalid neighboring points. Based on the obtained effective nearest neighbor points, calculate the average of the maximum height values ​​of all effective nearest neighbor points as the maximum height value of the missing point, and calculate the average of the minimum height values ​​of all effective nearest neighbor points as the minimum height value of the missing point. The corresponding height information is obtained by calculating the effective neighbor points, and the missing points are marked as effective points until all missing points are marked as effective points.

5. An apparatus for modeling an outdoor static environment based on a bird's eye view of a point cloud, characterized in that, include: The first module is used to acquire outdoor static environment point cloud data; The second module is used to perform point cloud coordinate system transformation based on the acquired outdoor static environment point cloud data to obtain the point cloud under vertical vision of the monitoring area, wherein the point cloud under vertical vision of the monitoring area is a world coordinate system with the z-axis perpendicular to the detection environment. The third module is used to generate a point cloud bird's-eye view based on the point cloud under the vertical vision of the monitored area. The fourth module is used to annotate the regional attributes of the point cloud bird's-eye view and obtain semantic information in the monitoring environment; The fifth module is used to generate an outdoor static area model based on the semantic information; The step of generating a point cloud bird's-eye view based on the point cloud under the vertical vision of the monitored area includes: Configure an origin point in the monitoring area, obtain the corresponding value of the origin point in the point cloud from the bird's-eye view, subtract the point cloud of the origin point from all the points in the point cloud, and convert the coordinate values ​​of all the point clouds in the monitoring area into positive values. Set the scale of each pixel in the bird's-eye view corresponding to the monitored area; Based on the established correspondence between pixels and real-world scale, and considering the different directions of the point cloud axis and the image axis, the pixel value corresponding to each point cloud in the monitoring environment when projected onto the bird's-eye view is calculated. The relationship between the pixels in the bird's-eye view and the point cloud is one-to-many. After obtaining the projection relationship between the point cloud and the bird's-eye view, the feature value of each pixel in the bird's-eye view is calculated to generate the point cloud bird's-eye view. The feature value includes the maximum height value and the minimum height value in the corresponding point cloud. Dynamic object extraction is performed based on the outdoor static area model, including: When a frame of point cloud data is obtained, the corresponding pixel points of all points in the corresponding frame are calculated on the bird's-eye view, and the height value of the pixel points is determined. Compare the height value of each pixel with the feature value of the corresponding pixel in the point cloud bird's-eye view; When the height value is outside the maximum and minimum height values ​​of the corresponding pixel, the corresponding pixel is a dynamic point, and the dynamic object is determined based on the dynamic point by the clustering method.

6. An electronic device, comprising: Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 4.

7. A computer readable storage medium characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images